Computing power arrangement method, device, equipment and storage medium

Abstract

The present application provides a computing power orchestration method, device, equipment and storage medium. It relates to the field of computer data processing technology. The method includes: virtualizing accelerator resources to achieve flexible and scalable task allocation; using an adaptive flow control mechanism to detect and manage contention between tasks; and coordinating resource allocation and contention management through an integrated runtime system. The present application ensures the achievement of service level agreements (SLAs) and performance stability by integrating adaptive accelerator allocation and dynamic contention management functions. In combination with virtualized accelerator orchestration and real-time contention regulation, accelerator and memory resources can be flexibly allocated, and DNN can be effectively deployed between multiple applications to ensure its efficient and reliable execution.

Description

Technical Field

[0001] The present application relates to the field of computer data processing technology, and in particular to a computing power scheduling method, device, equipment and storage medium. Background Art

[0002] With the rapid development of DNN applications in multiple fields, including edge computing and data centers, there is an increasing demand for parallel and efficient execution of multiple DNN workloads on shared hardware platforms. Existing solutions either suffer from inefficiency by statically partitioning resources or fail to cope with performance fluctuations caused by resource contention. A comprehensive approach is needed to optimize resource utilization and maintain performance stability while meeting the quality of service (QoS) requirements of co-resident DNN tasks. Summary of the invention

[0003] The present application provides a computing power scheduling method, device, equipment and storage medium to ensure the achievement of service level agreement (SLA) and performance stability by integrating adaptive accelerator allocation and dynamic contention management functions. Combined with virtualized accelerator scheduling and real-time contention adjustment, accelerators and memory resources can be flexibly allocated, DNNs can be effectively deployed between multiple applications, and their efficient and reliable execution can be guaranteed.

[0004] In a first aspect, the present application provides a computing power scheduling method, comprising:

[0005] Virtualize accelerator resources to achieve flexible and scalable task allocation;

[0006] Use adaptive flow control mechanisms to detect and manage contention between tasks;

[0007] Resource allocation and contention management are coordinated through an integrated runtime system.

[0008] Preferably, accelerator resources are virtualized to achieve flexible and scalable task allocation, including:

[0009] Abstracting physical accelerator resources into a flexibly allocated virtual resource pool, so that multiple tasks are not restricted to fixed physical accelerators, and resources can be dynamically acquired according to real-time needs;

[0010] Through the virtual interface, each task does not need to directly access a specific physical accelerator when requesting resources, but maps the task to a suitable virtual accelerator based on the current workload requirements and system resource status; wherein the virtualization interface is used to hide the specific location and status of the physical accelerator and simplify the interaction between the task and the accelerator;

[0011] Dynamically allocate virtual accelerators to each task based on task priority, latency requirements, and real-time resource availability.

[0012] Preferably, an adaptive flow control mechanism is used to detect and manage contention between tasks, including:

[0013] Detect resource contention points in real time based on the contention of memory bandwidth, accelerator computing units, and cache resources of each task;

[0014] Automatically adjust the memory access rate or execution rate of tasks according to resource contention points to achieve contention management; increase resource allocation for high-priority tasks; limit the access rate of low-priority tasks or non-critical tasks to ensure QoS for critical tasks;

[0015] Through the adaptive flow control mechanism, the memory access frequency of each task is dynamically adjusted according to the real-time feedback of the task, so as to ensure the performance stability of the task and avoid excessive contention of resources; wherein the real-time feedback includes delay and changes in bandwidth requirements.

[0016] Preferably, resource allocation and contention management are coordinated by an integrated runtime system, including:

[0017] The runtime system dynamically adjusts resource allocation strategies based on task load, system resource status, and contention to achieve resource optimization in a multi-tenant environment.

[0018] The state-adjusted resource allocation strategy includes two modes: balance mode and guarantee mode. In the balance mode, the system takes fairness as the goal and balances the resource allocation between tasks without special priority requirements, so that the SLA of all tasks can be basically met. In the guarantee mode, when there is a critical task that needs to meet the set QoS target, the system will give priority to allocating resources to the task, and at the same time appropriately limit the speed of non-critical tasks to ensure the service quality of critical tasks.

[0019] When the runtime system detects changes in resource usage or contention, it responds immediately and adjusts resource allocation strategies and flow control. Through efficient feedback and response mechanisms, it achieves dynamic adaptation to workloads and contention, and optimizes system performance in a multi-tenant environment.

[0020] In a second aspect, the present application provides a computing power scheduling device, the device comprising:

[0021] Virtualization accelerator management unit for flexible resource allocation;

[0022] Contention monitoring and regulation engine for adjusting memory access rates in real time;

[0023] Adaptive runtime system to synchronize resource allocation and contention policies based on workload demands.

[0024] Preferably, the virtualized accelerator management unit is further used to abstract the physical accelerator resources in the SoC and create a set of flexibly allocated virtual accelerators, so that multiple DNN tasks can share the same accelerator resources without being bound to a fixed physical accelerator. The virtualized accelerator management unit includes:

[0025] Dynamic resource pool management module, which is used to monitor the resource status of the current system, map tasks to suitable virtual accelerators according to the real-time requirements of tasks, and realize dynamic management and allocation of resource pools;

[0026] Task priority and demand matching module, which is used to dynamically adjust the allocated virtual resources according to the task priority, real-time bandwidth requirements and latency targets;

[0027] A virtualization interface is used to enable each task to directly access virtual accelerator resources without being bound to a specific physical accelerator.

[0028] Preferably, the contention monitoring and regulation engine comprises:

[0029] Contention monitoring module, which is used to monitor the occupation of system resources by each task in real time. It uses counters and registers to track the memory access pattern and bandwidth usage of tasks and identify possible contention areas in the system.

[0030] The dynamic contention detection module is used to detect and identify contention when the resource demand of a task increases or the system load changes, and report the current contention status to the adaptive runtime system in a timely manner through signal transmission to trigger corresponding adjustment measures;

[0031] The contention adjustment module is used to dynamically adjust resources based on the feedback results of the contention monitoring module. For critical tasks, their memory access rate or priority is increased, while for low-priority tasks, their bandwidth usage is limited. By adjusting the memory access rate and delay adjustment of tasks, the negative impact of contention on system performance can be reduced.

[0032] Preferably, the adaptive runtime system comprises:

[0033] A task scheduling module is used to allocate resources according to different priorities and resource requirements of tasks based on task scheduling strategies; wherein the task scheduling strategies include a guarantee strategy and a balance strategy, wherein the guarantee strategy is to prioritize the resource requirements of high-priority or low-latency tasks and appropriately limit the resource usage of low-priority tasks when system resources are tight, and the balance strategy is to balance resource usage by fair allocation for ordinary tasks without strict QoS requirements;

[0034] A real-time response module is used to adjust the current resource allocation and contention management strategy when the system resource usage status changes;

[0035] The feedback and optimization module is used to optimize future resource allocation plans based on the delayed feedback of tasks and resource usage patterns to improve the system's response speed and resource utilization efficiency.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the computing power orchestration method described in the first aspect and various possible designs of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the computing power orchestration method described in the first aspect and various possible designs of the first aspect is implemented.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the computing power orchestration method described in the first aspect and various possible designs of the first aspect.

[0039] The computing power orchestration method, apparatus, device and storage medium provided in this application can deploy scalable accelerators for multi-tenant workloads, support virtualized accelerator orchestration by jointly designing the accelerator software and hardware stack, thereby allowing the current workload to be adaptively bound to the available accelerators. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 A structural diagram of an integrated DNN multi-tenant execution system provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of a computing power scheduling method provided in an embodiment of the present application;

[0043] Figure 3 A flowchart of accelerator resource virtualization provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of contention between tasks managed by an adaptive flow control mechanism provided in an embodiment of the present application;

[0045] Figure 5 A flowchart of coordinating resource allocation and contention management through an integrated runtime system provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of the computing power orchestration device provided in an embodiment of the present application.

[0047] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0049] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0050] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0051] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0052] Computing power orchestration refers to the process of scheduling and managing computing power resources. Its core lies in dynamically allocating and adjusting computing power resources according to user needs through intelligent means to ensure efficient utilization and optimize costs. Computing power orchestration is an important part of the computing power network. Through orchestration and scheduling, flexible management and optimization of computing power resources can be achieved.

[0053] In the existing computing power scheduling method, for queue-based, first-come-first-served accelerator scheduling, the task thread will request its target number of accelerators to meet the QoS requirements by directly binding the accelerator. If a scheduling conflict occurs, that is, when there are fewer accelerators available in the system, an attempt will be made to pin the accelerator from another thread that will complete the current layer blocking the earliest, and then start execution after synchronization and adjustment of accelerator affinity. Since the user thread that pins the accelerator is used by physical accelerator binding, significant overhead is caused when reallocating accelerators.

[0054] Based on this, an embodiment of the present application provides a computing power orchestration method. The computing power orchestration method is a new full-stack computing power orchestration method in an extensible manner, which aims to solve the problems of low resource utilization efficiency and unstable task execution performance caused by inflexible accelerator resource allocation and limited contention adjustment capabilities in the existing multi-tenant DNN execution environment. The method realizes flexible and efficient allocation of accelerator resources in a multi-tenant environment through virtualized accelerator resources, adaptive flow control mechanism and unified scheduling system, and ensures the quality of service (QoS) and system throughput.

[0055] Figure 1 This is a structural diagram of the integrated DNN multi-tenant execution system provided in the embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides an integrated DNN multi-tenant execution system, and the computing power orchestration method is implemented based on the integrated DNN multi-tenant execution system. The integrated DNN multi-tenant execution system includes a virtualized accelerator layer 101, a contention-aware management layer 102, and an integrated scheduling and runtime system 103: the virtualized accelerator layer 101 supports flexible and real-time allocation of accelerator resources between multiple DNN workloads. Through the virtualization interface, the system optimizes the SLA achievement rate while ensuring the overall throughput. The contention-aware management layer 102 monitors system-level resource usage and applies an adaptive flow control mechanism to manage memory contention. It dynamically adjusts the memory access rate and processing rate according to the QoS requirements of the task, reduces interference between tasks, and improves the stability of system performance. The integrated scheduling and runtime system 103 is used as a central component to coordinate accelerator allocation and contention management. The integrated scheduling and runtime system 103 determines the optimal resource allocation and contention mitigation strategy in real time based on workload requirements and current contention levels.

[0056] Through synchronous accelerator allocation and contention management, the integrated DNN multi-tenant execution system supports efficient DNN deployment for a variety of delay-sensitive applications. Compared with traditional independent methods, the present invention significantly improves the system's throughput, fairness, and performance stability.

[0057] Specifically, it is reflected in the following four points:

[0058] First, virtualized accelerator allocation: The system uses virtualized accelerator orchestration technology to abstract physical accelerators into virtual entities. This virtualization allows multiple applications to flexibly share accelerator resources without static binding, greatly improving the scalability and resource utilization of the system.

[0059] Second, contention-aware management mechanism: The system integrates lightweight hardware and runtime to perform real-time contention detection and regulation. By monitoring memory access and adjusting traffic levels, the system can control interference caused by memory occupancy, ensure that the QoS requirements of high-priority tasks are achieved, and effectively share resources.

[0060] Third, unified scheduling and runtime: The comprehensive runtime monitors the resource requirements of each DNN task. It allocates accelerators, balances contention, and applies customized strategies based on latency and bandwidth requirements. The scheduler makes decisions between "balanced" and "guaranteed" contention strategies based on task priority and system load, thereby efficiently managing memory bandwidth and computing resources.

[0061] Fourth, performance adaptive algorithms: The system uses adaptive algorithms to evaluate and adjust task resource requirements. For example, when delay-sensitive applications may not be reached due to contention, the system can limit low-priority tasks and reallocate resources to maintain QoS.

[0062] The integrated DNN multi-tenant execution system has the following application scenarios and scalability:

[0063] 1. Suitable for multi-task execution in edge computing and data centers: The system is particularly suitable for scenarios such as edge computing devices and data centers that require multi-task parallel execution. In these scenarios, multiple DNN tasks need to share resources, and the system can ensure the QoS and performance stability of each task.

[0064] 2. Support for large-scale DNN deployment: The system's virtualized accelerator orchestration and dynamic contention-aware management modules provide scalability support for large-scale DNN tasks. Through resource virtualization and real-time monitoring, it can effectively cope with the rapid expansion needs of DNN tasks.

[0065] 3. Real-time applications in smart devices: The system is suitable for smart devices that require high performance and real-time response, such as autonomous driving, smart monitoring, and augmented reality. These applications need to process a large number of DNN inference tasks, and the system's contention management function ensures performance stability under high load.

[0066] In summary, the embodiments of the present application provide an efficient, fair and scalable multi-tenant DNN execution system. By integrating virtualized accelerator orchestration and dynamic contention-aware management, the system achieves flexible resource allocation and contention management, thereby ensuring the service quality of key tasks in a multi-tenant environment, optimizing resource utilization, and ensuring the overall performance stability of the system.

[0067] The computing power orchestration method provided in the embodiment of the present application can be configured and implemented in the system described above, and can be used for multi-tenant DNN workload accelerator resource orchestration. It can integrate dynamic orchestration and contention management. Through the combination of the two, the resource utilization of DNN multi-tenant execution on a shared hardware platform is improved, strict QoS goals are met, and delay fluctuations are reduced, which is of great significance for real-time applications. Figure 2 This is a flow chart of the computing power scheduling method provided in the embodiment of the present application. Figure 2 As shown, the computing power scheduling method includes the following steps S210 to S230.

[0068] S210: Virtualize accelerator resources to achieve flexible and scalable task allocation.

[0069] In some embodiments, Figure 3 FIG. 2 is a flowchart of accelerator resource virtualization provided in an embodiment of the present application. Step S210 is specifically implemented by the following steps:

[0070] S211: Definition of virtualization: Virtualization of accelerator resources refers to abstracting physical accelerator resources into a flexibly allocated virtual resource pool, so that multiple tasks are not restricted to fixed physical accelerators and can dynamically obtain resources according to real-time needs.

[0071] S212: Implementation of virtualization: Through the virtual interface, each task does not need to directly access a specific physical accelerator when requesting resources. Instead, the system maps the task to a suitable virtual accelerator based on the current workload requirements and system resource status. The virtualization interface can hide the specific location and status of the physical accelerator and simplify the interaction between the task and the accelerator.

[0072] S213: Task allocation mechanism: The system dynamically allocates virtual accelerators to each task based on the task priority, latency requirements, and real-time resource conditions. In this way, the system can flexibly adjust the resource allocation of tasks based on the task service level agreement (SLA), ensure the balance of resources between tasks, and avoid resource waste caused by static resource binding.

[0073] S220: Use an adaptive flow control mechanism to detect and manage contention between tasks.

[0074] In some embodiments, Figure 4As shown, it is a flowchart of the adaptive flow control mechanism management task contention provided by the embodiment of the present application. Step S220 is specifically implemented by the following steps:

[0075] S221: Contention detection: The system monitors the resource usage of each task, especially the contention of memory bandwidth, accelerator computing unit and cache resources. By monitoring the memory access rate and bandwidth usage of each task, the system can detect resource contention points in real time.

[0076] S222: Adaptive Adjustment: Based on the contention situation, the system automatically adjusts the memory access rate or execution rate of the task to achieve contention management. For high-priority tasks, the system can increase resource allocation; for low-priority tasks or non-critical tasks, the system will limit their access rate to ensure the QoS of critical tasks.

[0077] S223: Adaptive flow control mechanism, which dynamically adjusts the memory access frequency of each task based on real-time feedback from the task (such as latency, changes in bandwidth requirements, etc.) to ensure the performance stability of the task and avoid excessive resource contention.

[0078] S230: Coordinate resource allocation and contention management through an integrated runtime system.

[0079] In some embodiments, the runtime system is the core of the entire accelerator orchestration and contention management. It plays a global coordination role in task scheduling, resource allocation, contention detection and regulation. The runtime system can dynamically adjust resource allocation strategies according to task load, system resource status and contention to achieve resource optimization in a multi-tenant environment. Figure 5 As shown, it is a flowchart of coordinating resource allocation and contention management through an integrated runtime system provided by an embodiment of the present application, and step S230 is specifically implemented by the following steps:

[0080] S231: Execute the scheduling strategy, which includes two types: "balance mode" and "guarantee mode".

[0081] Balanced mode: In the absence of special priority requirements, the system takes fairness as the goal and balances the resource allocation between tasks so that the SLA of all tasks can be basically met.

[0082] Guarantee mode: When a critical task needs to meet a specific QoS target, the system will prioritize the allocation of resources to the task, while appropriately limiting the speed of non-critical tasks to ensure the service quality of the critical task.

[0083] S232: Execute real-time response mechanism: After detecting changes in resource usage or contention, the system can respond immediately and adjust resource allocation strategies and flow control. Through efficient feedback and response mechanisms, the system achieves dynamic adaptation to workloads and contention, and optimizes system performance in a multi-tenant environment.

[0084] In summary, the computing power orchestration method provided in the embodiment of the present application can realize flexible, efficient and contention-aware accelerator resource management, provide good performance support for the multi-tenant DNN execution environment, and ensure the service quality and fairness of the system under high load, which is specifically reflected in the following three points:

[0085] First, flexible resource utilization: By virtualizing accelerator resources, the resource waste caused by static binding is avoided, and flexible resource allocation in a multi-tenant DNN environment is achieved.

[0086] Second, performance stability: The adaptive flow control mechanism ensures performance stability when multiple tasks are executed concurrently, reducing delay fluctuations of high-priority tasks due to contention.

[0087] Third, efficient service quality assurance: Through the integrated runtime system, the resource allocation of different tasks can be effectively managed to ensure that the service quality of key tasks is not affected, while improving the resource utilization and throughput of the overall system.

[0088] The embodiment of the present application also provides a computing power orchestration device, which is an efficient hardware device for implementing resource management and contention control in a multi-tenant deep neural network (DNN) execution environment on a system-on-chip (SoC). The device is suitable for use in DNN multi-task concurrency scenarios, and through intelligent resource allocation and contention adjustment, it achieves efficient utilization of system resources, ensures fairness between different tasks and the quality of service (QoS) of key tasks. Figure 6 This is a schematic diagram of the structure of the computing power arrangement device provided in the embodiment of the present application. Figure 6 As shown, the computing power arrangement device includes:

[0089] A virtualization accelerator management unit 610, used for flexibly allocating resources;

[0090] Contention monitoring and regulation engine 620, for adjusting memory access rate in real time;

[0091] Adaptive runtime system 630 is used to synchronize resource allocation and contention strategies according to workload requirements.

[0092] In some embodiments, the virtualized accelerator management unit 610 is used to abstract the physical accelerator resources in the SoC and create a set of flexibly assignable virtual accelerators, so that multiple DNN tasks can share the same accelerator resources without being bound to a fixed physical accelerator. Figure 6 As shown, the virtualization accelerator management unit 610 includes a dynamic resource pool management module 611 , a task priority and demand matching module 612 , and a virtualization interface 613 .

[0093] For the dynamic resource pool management module 611, the virtualized accelerator management unit creates an accelerator resource pool and allocates virtual accelerator instances when tasks request resources. The management unit can monitor the resource status of the current system and map tasks to suitable virtual accelerators according to the real-time requirements of the tasks, thereby realizing dynamic management and allocation of the resource pool.

[0094] The task priority and demand matching module 612 can dynamically adjust the allocated virtual resources according to the task priority, real-time bandwidth demand and delay target. For example, for tasks with high priority or low delay demand, the system can give priority to allocating more computing resources to ensure that they meet the QoS requirements.

[0095] The virtualized accelerator management unit 610 enables each task to directly access the virtual accelerator resources without being bound to a specific physical accelerator via the virtualized interface 613. The design of the virtualized interface 613 simplifies the task scheduling process and makes resource allocation more flexible and scalable.

[0096] In some embodiments, Figure 6 As shown, the contention monitoring and regulation engine 620 includes:

[0097] Contention monitoring module 621: This module monitors the usage of system resources by each task in real time, especially the usage of memory bandwidth and cache resources. The monitoring module uses counters and registers to track the memory access pattern and bandwidth usage of tasks and identify possible contention areas in the system.

[0098] Dynamic contention detection module 622: When the resource demand of a task increases or the system load changes, causing the contention to intensify, the contention monitoring module can quickly detect and identify the contention. The system can report the current contention status to the adaptive runtime system in a timely manner through signal transmission to trigger corresponding adjustment measures.

[0099] Contention adjustment module 623: Based on the feedback from the contention monitoring module, the contention adjustment module dynamically adjusts resources. For example, for critical tasks, the system can increase their memory access rate or priority, while for low-priority tasks, their bandwidth usage will be limited. The adjustment module can reduce the negative impact of contention on system performance by adjusting the memory access rate of tasks, delay adjustment, etc.

[0100] The contention adjustment module 623 has a real-time adjustment mechanism. The contention adjustment module optimizes resource allocation through an adaptive algorithm according to the contention status monitored in real time to ensure the overall stability of the system. By increasing or decreasing the resource usage frequency of tasks in a timely manner, the interference between tasks is minimized to meet the performance requirements of key tasks.

[0101] In some embodiments, the adaptive runtime system 630 is the core management module of the computing power orchestration device, responsible for scheduling and allocating resources, and coordinating the work of each functional unit according to real-time task requirements and contention. The adaptive runtime system 630 can dynamically analyze the performance requirements of tasks and adjust resource allocation and contention management strategies.

[0102] like Figure 6 As shown, the adaptive runtime system 630 includes a task scheduling module 631, a real-time response module 632 and a feedback and optimization module 633:

[0103] The task scheduling module 631 is used to execute task scheduling strategies. The task scheduling module 631 includes multiple task scheduling strategies and can perform appropriate resource allocation according to different priorities and resource requirements of tasks.

[0104] For example, task scheduling strategies include:

[0105] Guarantee strategy: For high-priority or low-latency tasks, the adaptive runtime system prioritizes their resource requirements and appropriately limits the resource usage of low-priority tasks when system resources are tight.

[0106] Balance strategy: For common tasks without strict QoS requirements, the adaptive runtime system will adopt a fair allocation approach to balance resource usage and prevent individual tasks from affecting the overall system performance due to contention.

[0107] The real-time response module 632 is used to implement the real-time response mechanism. When the system resource usage status changes, the adaptive runtime system can respond quickly and adjust the current resource allocation and contention management strategy. For example, when it is detected that the system load increases and the key task is disturbed, the system will immediately prioritize the allocation of resources to the key task.

[0108] The feedback and optimization module 633 continuously optimizes resource allocation decisions based on the execution feedback of the current task. For example, the system will optimize future resource allocation plans based on the delayed feedback of the task and the resource usage pattern to improve the system's response speed and resource utilization efficiency.

[0109] In summary, the computing power arrangement device provided in the embodiment of the present application has the following beneficial effects:

[0110] 1) Flexibility in resource utilization: Through the virtualized accelerator management unit, the system can realize flexible resource allocation, avoid resource waste caused by static binding, and improve the adaptability of SoC in a multi-tenant environment.

[0111] 2) Contention management effectiveness: The contention monitoring and adjustment engine can detect resource contention problems in a timely manner and quickly adjust the resource usage of tasks, ensuring the stable performance of the system in a high-concurrency environment, especially the QoS achievement rate of key tasks.

[0112] 3) Efficient runtime scheduling: The adaptive runtime system has real-time response and intelligent adjustment capabilities.

[0113] An embodiment of the present application provides an electronic device, which may include: a processor and a memory, wherein the processor and the memory may communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0114] The processor executes the computer execution instructions stored in the memory, so that the processor executes the scheme in the above embodiment. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0115] The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0116] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0117] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the computing power orchestration method of the above embodiment.

[0118] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the computing power orchestration method in the above embodiment can be implemented.

[0119] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0120] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0121] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0122] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0123] It should be understood that the above processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0124] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0125] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0126] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0127] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0128] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A computing power scheduling method, characterized in that: The method comprises: Virtualize accelerator resources to achieve flexible and scalable task allocation; Use adaptive flow control mechanisms to detect and manage contention between tasks; Coordinate resource allocation and contention management through an integrated runtime system; Virtualize accelerator resources to enable flexible and scalable task allocation, including: Abstract the physical accelerator resources into a flexibly allocated virtual resource pool. Abstract the physical accelerator resources within the SoC and create a set of flexibly allocated virtual accelerators, so that multiple DNN tasks can share the same accelerator resources without being bound to a fixed physical accelerator. Through the virtualization interface, each task does not need to directly access a specific physical accelerator when requesting resources, but instead maps the task to a suitable virtual accelerator based on the current workload requirements and system resource status; wherein the virtualization interface is used to hide the specific location and status of the physical accelerator and simplify the interaction between the task and the accelerator; Dynamically allocate virtual accelerators to each task based on the task's priority, latency requirements, and real-time resource availability; Resource allocation and contention management are coordinated through an integrated runtime system, including: The runtime system dynamically adjusts resource allocation strategies based on task load, system resource status, and contention to achieve resource optimization in a multi-tenant environment. The dynamic resource allocation strategy includes two modes: balance mode and guarantee mode. In the balance mode, the system takes fairness as the goal and balances the resource allocation between tasks without special priority requirements, so that the SLA of all tasks can be basically met. In the guarantee mode, when there is a critical task that needs to meet the set QoS target, the system will give priority to allocating resources to the task, and at the same time appropriately limit the speed of non-critical tasks to ensure the service quality of critical tasks. When the runtime system detects changes in resource usage or contention, it responds immediately and adjusts resource allocation strategies and flow control. Through efficient feedback and response mechanisms, it achieves dynamic adaptation to workloads and contention, and optimizes system performance in a multi-tenant environment.

2. The method according to claim 1, characterized in that: Use adaptive flow control mechanisms to detect and manage contention between tasks, including: Detect resource contention points in real time based on the contention of memory bandwidth, accelerator computing units, and cache resources of each task; Automatically adjust the memory access rate or execution rate of tasks according to resource contention points to achieve contention management; increase resource allocation for high-priority tasks; limit the access rate of low-priority tasks or non-critical tasks to ensure QoS for critical tasks; Through the adaptive flow control mechanism, the memory access frequency of each task is dynamically adjusted according to the real-time feedback of the task, so as to ensure the performance stability of the task and avoid excessive contention of resources; wherein the real-time feedback includes delay and changes in bandwidth requirements.

3. A computing power arrangement device, characterized in that: The device comprises: A virtualized accelerator management unit, used to virtualize accelerator resources to achieve flexible and scalable task allocation; Contention monitoring and regulation engine for detecting and managing contention between tasks using adaptive flow control mechanisms; Adaptive runtime system for coordinating resource allocation and contention management through an integrated runtime system; The virtualization accelerator management unit is also used for: Abstract the physical accelerator resources into a flexibly allocated virtual resource pool. Abstract the physical accelerator resources within the SoC and create a set of flexibly allocated virtual accelerators, so that multiple DNN tasks can share the same accelerator resources without being bound to a fixed physical accelerator. Through the virtualization interface, each task does not need to directly access a specific physical accelerator when requesting resources, but instead maps the task to a suitable virtual accelerator based on the current workload requirements and system resource status; wherein the virtualization interface is used to hide the specific location and status of the physical accelerator and simplify the interaction between the task and the accelerator; Dynamically allocate virtual accelerators to each task based on the task's priority, latency requirements, and real-time resource availability; The adaptive runtime system is also used to: The runtime system dynamically adjusts resource allocation strategies based on task load, system resource status, and contention to achieve resource optimization in a multi-tenant environment. The dynamic resource allocation strategy includes two modes: balance mode and guarantee mode. In the balance mode, the system takes fairness as the goal and balances the resource allocation between tasks without special priority requirements, so that the SLA of all tasks can be basically met. In the guarantee mode, when there is a critical task that needs to meet the set QoS target, the system will give priority to allocating resources to the task, and at the same time appropriately limit the speed of non-critical tasks to ensure the service quality of critical tasks. When the runtime system detects changes in resource usage or contention, it responds immediately and adjusts resource allocation strategies and flow control. Through efficient feedback and response mechanisms, it achieves dynamic adaptation to workloads and contention, and optimizes system performance in a multi-tenant environment.

4. The computing power arrangement device according to claim 3, characterized in that: The contention monitoring and regulation engine includes: Contention monitoring module, which is used to monitor the occupation of system resources by each task in real time. It uses counters and registers to track the memory access pattern and bandwidth usage of tasks and identify possible contention areas in the system. The dynamic contention detection module is used to detect and identify contention when the resource demand of a task increases or the system load changes, and report the current contention status to the adaptive runtime system in a timely manner through signal transmission to trigger corresponding adjustment measures; The contention adjustment module is used to dynamically adjust resources based on the feedback results of the contention monitoring module. For critical tasks, their memory access rate or priority is increased, while for low-priority tasks, their bandwidth usage is limited. By adjusting the memory access rate and delay adjustment of tasks, the negative impact of contention on system performance can be reduced.

5. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 2 when executed by a processor.

Images