Asynchronous scheduling method, device and equipment of heterogeneous computing task and storage medium

Abstract

The application discloses a kind of asynchronous scheduling methods, device and equipment of heterogeneous computing task and storage medium, involve artificial intelligence evaluation technical field, comprising: obtaining the heterogeneous computing task to be processed, the resource requirement type of heterogeneous computing task is obtained by parsing heterogeneous computing task, and according to resource requirement type, heterogeneous computing task is distributed to the asynchronous processing queue on target computing node in multi-node cluster environment;In asynchronous processing queue, the preset routing voucher corresponding to heterogeneous computing task is generated, and preset routing voucher is circulated;Based on the task identifier of preset routing voucher in circulation, obtain the running context of heterogeneous computing task, and issue the execution instruction of current stage to target execution environment;In response to the execution instruction of current stage execution completion, according to the task attribute information of heterogeneous computing task, determine the asynchronous processing queue of next stage from multiple asynchronous processing queues, and asynchronously route preset routing voucher to the asynchronous processing queue of next stage.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence evaluation technology, and in particular to asynchronous scheduling methods, apparatus, devices and storage media for heterogeneous computing tasks. Background Technology

[0002] Currently, complex AI evaluation systems, such as embodied intelligence, typically need to handle multiple evaluation scenarios with vastly different resource characteristics simultaneously, such as pure inference evaluation, code execution evaluation, and multimodal perception evaluation. Existing solutions usually employ a single pipeline with a shared resource pool or a unified sandbox lifecycle management strategy to handle these heterogeneous workloads. However, because different evaluation stages have different requirements for the type of computing resources, concurrency characteristics, and lifecycle, shared resource pools can easily lead to slow stages blocking fast stages, resulting in low overall resource utilization; unified sandbox management cannot simultaneously meet the differentiated needs of task-level sharing and round-level exclusive access; and the lack of effective deployment rate control and orphan container recycling mechanisms can easily trigger deployment storms or long-term resource leaks due to process anomalies during large-scale multi-node deployments. Summary of the Invention

[0003] The main objective of this application is to provide an asynchronous scheduling method, apparatus, device, and storage medium for heterogeneous computing tasks, aiming to solve the technical problems of low scheduling efficiency, chaotic resource lifecycle management, and poor multi-node scalability in existing solutions.

[0004] To achieve the above objectives, this application proposes an asynchronous scheduling method for heterogeneous computing tasks, the asynchronous scheduling method for heterogeneous computing tasks comprising: Obtain the heterogeneous computing task to be processed, parse the task attribute information of the heterogeneous computing task to obtain the resource requirement type of the heterogeneous computing task, and allocate the heterogeneous computing task to the asynchronous processing queue on the target computing node in the multi-node cluster environment according to the resource requirement type. In the asynchronous processing queue, a preset routing credential corresponding to the heterogeneous computing task is obtained in an asynchronous waiting manner, and the preset routing credential is transferred. The preset routing credential includes at least a task identifier. Based on the task identifier of the preset routing credential in the flow, the execution context of the heterogeneous computing task is obtained from the global state storage area, and the execution instruction of the current stage is sent to the target execution environment according to the execution context. In response to the completion of the execution instruction of the current stage, the asynchronous processing queue for the next stage is determined from multiple asynchronous processing queues based on the task attribute information of the heterogeneous computing task, and the preset routing credential is asynchronously routed to the asynchronous processing queue of the next stage.

[0005] In one embodiment, the steps of acquiring the heterogeneous computing task to be processed, parsing the task attribute information of the heterogeneous computing task to obtain the resource requirement type of the heterogeneous computing task, and allocating the heterogeneous computing task to the asynchronous processing queue on the target computing node in a multi-node cluster environment according to the resource requirement type include: Obtain the heterogeneous computing tasks to be processed, parse the task attribute information of the heterogeneous computing tasks, and obtain the resource requirement type of the heterogeneous computing tasks; Based on the resource requirement type, a target computing node is determined from the multi-node cluster. The resource requirement type includes at least one of graphics processor inference resources, central processing unit computing resources, exclusive sandbox resources, and shared sandbox resources. Obtain the real-time resource level of each computing node in the multi-node cluster corresponding to the resource demand type; Based on the real-time resource level, a greedy binning algorithm is used to select candidate computing nodes from the computing nodes that meet the resource requirements of the heterogeneous computing task. Obtain the backlog of jobs to be run on each candidate computing node, and determine the candidate computing node whose backlog does not exceed a preset back pressure threshold as the target computing node; If the backlog exceeds the preset back pressure threshold, an independent orchestration instance is triggered to perform elastic scaling on the candidate compute nodes, and the scaled-up candidate compute nodes are determined as the target compute nodes. On the target computing node, the heterogeneous computing tasks are assigned to asynchronous processing queues corresponding to the resource requirement types.

[0006] In one embodiment, after the step of triggering an independent orchestration instance to perform elastic scaling on the candidate compute nodes when the backlog exceeds the preset back pressure threshold, and determining the scaled-up candidate compute nodes as the target compute nodes, the method further includes: The resource usage status of the target computing node is monitored through a separately running monitoring thread; If the resource occupancy status is detected to have reached a preset full load threshold and there is an external task request with a priority higher than the current priority to access the target node, the historical heterogeneous computing tasks submitted to the target computing node are automatically canceled to release the computing resources of the target computing node for external task scheduling.

[0007] In one embodiment, the step of determining the next asynchronous processing queue from multiple asynchronous processing queues based on the task attribute information of the heterogeneous computing task, and asynchronously routing the preset routing credential to the next asynchronous processing queue in response to the completion of the execution instruction of the current stage, includes: In response to the completion of the execution instruction of the current stage, the resource attributes and isolation allocation mode configured in the task attribute information of the heterogeneous computing task are obtained; Based on the resource attributes and the isolation allocation mode, the next asynchronous processing queue is determined from multiple asynchronous processing queues, and the preset routing credentials are asynchronously routed to the next asynchronous processing queue.

[0008] In one embodiment, the step of determining the next-stage asynchronous processing queue from multiple asynchronous processing queues based on the resource attributes and the isolation allocation mode, and asynchronously routing the preset routing credentials to the next-stage asynchronous processing queue includes: If the resource attribute indicates that a decision service is required, the asynchronous processing queue for the next stage is determined as the decision service queue, and the preset routing credentials are routed to the decision service queue. If the resource attribute indicates that computing services are required, the asynchronous processing queue for the next stage is determined to be the local computing queue, and the preset routing credentials are routed to the local computing queue. When the resource attribute indicates that an isolated execution environment is required and the isolation allocation mode is task-level sharing, the asynchronous processing queue for the next stage is determined to be the shared isolated execution environment queue, and the preset routing credentials are asynchronously routed to the shared isolated execution environment queue. If the resource attribute indicates that an isolated execution environment is required and the isolation allocation mode is round-based exclusive, the asynchronous processing queue for the next stage is determined to be the exclusive isolated execution environment queue, and the preset routing credential is asynchronously routed to the exclusive isolated execution environment queue.

[0009] In one embodiment, the method further includes: During the execution of the heterogeneous computing task, a lease log is maintained in the global state storage area in an append-only manner. The lease log is used to record lease acquisition events and lease release events of the resources requested by the heterogeneous computing task. The lease log is periodically scanned, and if an abnormal task is identified that has a lease acquisition event but no corresponding lease release event is recorded, the execution resources occupied by the abnormal task are reclaimed. Before the heterogeneous computing task is executed, an atomic renaming operation is performed on the lease log to generate a snapshot file. Based on the snapshot file, historical tasks that did not end normally are identified, and the execution resources occupied by the historical tasks are reclaimed.

[0010] In one embodiment, the step of sending the execution instructions for the current stage to the target execution environment according to the running context includes: Based on the runtime context, a request to obtain sandbox resources is initiated to the sandbox manager, and the sandbox resource instance returned by the sandbox manager is received. Based on the isolation allocation mode corresponding to the heterogeneous computing task, the corresponding handle protocol is used to manage the lifecycle of the sandbox resource instance; During the process of acquiring the sandbox resource instance, a gating semaphore is deployed to perform concurrent rate limiting on the requests of the sandbox manager to create sandbox resources, in order to suppress deployment storms; The sandbox resource instance is used as the target execution environment, and the execution instructions for the current stage are sent to the target execution environment.

[0011] In one embodiment, after the step of initiating a request to the sandbox manager to obtain sandbox resources based on the runtime context and receiving the sandbox resource instance returned by the sandbox manager, the method further includes: Based on the isolation allocation mode corresponding to the heterogeneous computing task, the corresponding handle protocol is used to manage the lifecycle of the sandbox resource instance; During the process of acquiring the sandbox resource instance, a gating semaphore is deployed to perform concurrent rate limiting on the requests of the sandbox manager to create sandbox resources, in order to suppress deployment storms.

[0012] In one embodiment, the step of managing the lifecycle of the sandbox resource instance using the corresponding handle protocol based on the isolation allocation mode corresponding to the heterogeneous computing task includes: When the isolation allocation mode corresponding to the heterogeneous computing task is task-level sharing, the concurrent control of the sandbox resource instance is performed through asynchronous locks and reference counting mechanism, and the sandbox resource instance is kept alive when the reference count of the sandbox resource instance is greater than zero. When the isolation allocation mode corresponding to the heterogeneous computing task is round-based exclusive, the sandbox resource instances are concurrently controlled by an activity gating semaphore to limit the number of exclusive sandbox resource instances that are running at the same time.

[0013] Furthermore, to achieve the above objectives, this application also proposes an asynchronous scheduling device for heterogeneous computing tasks, the asynchronous scheduling device for heterogeneous computing tasks comprising: The acquisition module is used to acquire heterogeneous computing tasks to be processed, parse the task attribute information of the heterogeneous computing tasks, obtain the resource requirement type of the heterogeneous computing tasks, and allocate the heterogeneous computing tasks to the asynchronous processing queues on the target computing nodes in a multi-node cluster environment according to the resource requirement type. A generation module is used to obtain a preset routing credential corresponding to the heterogeneous computing task in the asynchronous processing queue in an asynchronous waiting manner, and to transfer the preset routing credential, wherein the preset routing credential includes at least a task identifier; The acquisition module is further configured to acquire the running context of the heterogeneous computing task from the global state storage area based on the task identifier of the preset routing credential in the flow, and send the execution instruction of the current stage to the target execution environment according to the running context; The determination module is used to respond to the completion of the execution instruction of the current stage, determine the asynchronous processing queue of the next stage from multiple asynchronous processing queues according to the task attribute information of the heterogeneous computing task, and asynchronously route the preset routing credential to the asynchronous processing queue of the next stage.

[0014] Furthermore, to achieve the above objectives, this application also proposes an asynchronous scheduling device for heterogeneous computing tasks, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the asynchronous scheduling method for heterogeneous computing tasks as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the asynchronous scheduling method for heterogeneous computing tasks as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the asynchronous scheduling method for heterogeneous computing tasks as described above.

[0017] This application proposes one or more technical solutions to acquire heterogeneous computing tasks to be processed, parse the task attribute information of the heterogeneous computing tasks to obtain the resource requirement type of the heterogeneous computing tasks, and allocate the heterogeneous computing tasks to asynchronous processing queues on target computing nodes in a multi-node cluster environment according to the resource requirement type; in the asynchronous processing queues, acquire the preset routing credentials corresponding to the heterogeneous computing tasks in an asynchronous waiting manner, and circulate the preset routing credentials, which at least include a task identifier; based on the task identifier of the circulating preset routing credentials, acquire the running context of the heterogeneous computing tasks from the global state storage area, and send the execution instructions of the current stage to the target execution environment according to the running context; in response to the completion of the execution instructions of the current stage, determine the asynchronous processing queue of the next stage from multiple asynchronous processing queues according to the task attribute information of the heterogeneous computing tasks, and asynchronously route the preset routing credentials to the asynchronous processing queue of the next stage. By decoupling task scheduling from state storage, lightweight preset routing credentials are used to flow between queues, avoiding memory bloat and consistency risks caused by copying the complete context between queues. At the same time, the dynamic routing mechanism based on task attribute information enables tasks with different resource requirements to be automatically diverted to the corresponding dedicated queues, eliminating mutual blocking caused by heterogeneous resource sharing thread pools and improving the overall throughput and resource utilization of the system. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an embodiment of the asynchronous scheduling method for heterogeneous computing tasks in this application; Figure 2 This is a schematic diagram of the architecture of an asynchronous scheduling system for heterogeneous computing tasks provided in an embodiment of the asynchronous scheduling method for heterogeneous computing tasks in this application. Figure 3 A flowchart illustrating the second embodiment of the asynchronous scheduling method for heterogeneous computing tasks in this application; Figure 4 A flowchart illustrating the asynchronous scheduling method for heterogeneous computing tasks in this application (Example 3). Figure 5This is a schematic diagram of the module structure of the asynchronous scheduling device for heterogeneous computing tasks in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware operating environment involved in the asynchronous scheduling method for heterogeneous computing tasks in the embodiments of this application.

[0021] Explanation of icon numbers: Orchestrator 10, Sandbox Manager 20, Auto Deployment Circulator 30; The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: Obtain the heterogeneous computing task to be processed, parse the task attribute information of the heterogeneous computing task to obtain the resource requirement type of the heterogeneous computing task, and allocate the heterogeneous computing task to the asynchronous processing queue on the target computing node in a multi-node cluster environment according to the resource requirement type; in the asynchronous processing queue, obtain the preset routing credential corresponding to the heterogeneous computing task in an asynchronous waiting manner, and circulate the preset routing credential, which includes at least a task identifier; based on the task identifier of the circulating preset routing credential, obtain the running context of the heterogeneous computing task from the global state storage area, and send the execution instruction of the current stage to the target execution environment according to the running context; in response to the completion of the execution instruction of the current stage, determine the asynchronous processing queue of the next stage from multiple asynchronous processing queues according to the task attribute information of the heterogeneous computing task, and asynchronously route the preset routing credential to the asynchronous processing queue of the next stage.

[0025] In existing technologies, for heterogeneous computing task scheduling in multi-node cluster environments, a unified shared thread pool is usually used to handle tasks with different resource requirements. Tasks with different resource requirements are prone to blocking each other, and the task context needs to be completely copied when the task flows between different queues, which can easily lead to memory bloat and state consistency issues, resulting in low overall system throughput and low resource utilization.

[0026] This application provides a solution that decouples task scheduling from state storage. It utilizes lightweight, pre-defined routing credentials to replace the flow of complete task contexts between queues. The runtime context is retrieved from a globally unified state storage area only when task phase instructions need to be executed, using the task identifier in the credential. This fundamentally avoids the memory bloat caused by multiple copies of the complete context between queues and the risks of inconsistent state synchronization among multiple nodes. Furthermore, this solution establishes a dynamic routing mechanism based on task attribute information. It can automatically distribute tasks to dedicated asynchronous processing queues of corresponding types based on their resource requirements and isolation allocation modes. Each asynchronous queue is independently configured with dedicated execution resources. This architecturally eliminates the blocking problems between nodes and tasks caused by sharing thread pools for heterogeneous resource tasks of different types. It effectively improves the overall task processing throughput and heterogeneous resource utilization of multi-node cluster systems, adapting to the complex heterogeneous computing task scheduling needs in large-scale multi-node cluster environments.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, an asynchronous scheduling device for heterogeneous computing tasks, etc. The following description uses an asynchronous scheduling device for heterogeneous computing tasks as an example to illustrate this embodiment and the subsequent embodiments. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the corresponding device.

[0028] Based on this, embodiments of this application provide an asynchronous scheduling method for heterogeneous computing tasks, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the asynchronous scheduling method for heterogeneous computing tasks in this application.

[0029] In this embodiment, the asynchronous scheduling method for heterogeneous computing tasks includes steps S10 to S40: Step S10: Obtain the heterogeneous computing task to be processed, parse the task attribute information of the heterogeneous computing task to obtain the resource requirement type of the heterogeneous computing task, and allocate the heterogeneous computing task to the asynchronous processing queue on the target computing node in the multi-node cluster environment according to the resource requirement type.

[0030] It should be noted that heterogeneous computing tasks refer to a set of subtasks with significantly different resource requirements within a complete business flow. For example, in embodied intelligence evaluation, a single evaluation task may simultaneously include multiple types such as pure text reasoning, code sandbox execution, terminal interactive operation, and local CPU scoring. The core concept of this embodiment is to organize scheduling units based on "resource requirement type" rather than "pipeline stage" on top of a single asynchronous runtime. The execution stage is subdivided into independent queues according to its required resources (decision service, shared sandbox, exclusive sandbox, CPU), and each queue is equipped with a worker pool with an independent concurrency budget. Unlike traditional technologies where all stages share a single resource pool or execute in a fixed pipeline order, the asynchronous processing queues in this embodiment are divided according to resource requirement type. For example, the system constructs multiple independent queues such as a reset queue, a generation queue, a decision queue, a shared sandbox queue, an exclusive sandbox queue, a CPU computation queue, and an output queue. Each queue specifically serves a particular type of resource requirement and is equipped with independent concurrency control parameters. This design allows I / O-intensive inference tasks and computationally intensive sandbox tasks to flow independently in their respective queues, eliminating mutual blocking caused by resource mismatch and significantly improving the overall throughput of the system.

[0031] In practice, asynchronous asyncio.Queue can be replaced with a multi-process queue, a distributed message queue (such as Redis Stream, Kafka), or an actor model mailbox, while the queue topology and routing rules remain unchanged.

[0032] Step S20: In the asynchronous processing queue, obtain the preset routing credential corresponding to the heterogeneous computing task in an asynchronous waiting manner, and transfer the preset routing credential, which includes at least a task identifier.

[0033] Understandably, the preset routing credential is also referred to as an "envelope" in this embodiment. It is a lightweight data carrier that contains only necessary routing information such as the task identifier (ctx_id) and the current processing phase identifier, with a data size typically around 100 bytes. In contrast, the complete runtime context contains a large amount of state information such as input data, intermediate results, and observations, typically ranging from 10KB to 50KB in size. This embodiment adopts the design principle of "envelope-context" separation, that is, only the lightweight preset routing credential flows between queues, while the heavy runtime context is centrally stored in the global state storage area. During the flow, the producer of the generation queue or other upstream queues writes the preset routing credential to the message middleware or asynchronous queue and returns immediately without waiting for the downstream consumer to complete processing, thus achieving true asynchronous non-blocking.

[0034] In one feasible implementation, the preset routing credential is presented as a minimal data structure, encapsulating only the generated task identifier (ctx_id) and the phase identifier representing the current business flow position. Its serialized data size is typically only around 100 bytes. Lightweight envelopes are used to pass references to evaluation rounds between queues, and the complete state is centrally stored. The complete runtime context contains a large amount of state data, including input samples, historical observation records, and model response caches, often reaching 10KB to 50KB or even higher in size. After constructing this lightweight credential, the orchestrator writes it as a message to a specified topic in a message broker (such as Redis Stream, Kafka, or RabbitMQ). The producer thread returns immediately after confirming that the message has been persisted or received by the broker, without waiting for downstream consumers to have read or processed the message. For example, after the generation queue completes a large model inference, it only needs to push a small data packet containing ctx_id and phase='step_judge' into the Topic named 'task_routing', which takes a short time. If the traditional approach is used to pass the complete context, it may take a lot of time due to network serialization and transmission. This design physically severs the strong coupling between the upstream production rate and the downstream consumption rate, enabling the system to use the buffering capacity of the message middleware to smooth out peaks and valleys when instantaneous traffic surges arrive, avoiding the risk of upstream threads being blocked or even the entire scheduler crashing due to slow downstream processing.

[0035] Understandably, each asynchronous processing queue's corresponding Worker (worker process) runs an independent listener coroutine or thread. These listeners, as members of a Consumer Group, continuously subscribe to message middleware topics. When a new message arrives, the listener retrieves the preset routing credentials via asynchronous pull or push callbacks, parses out the task and stage identifiers, and then triggers subsequent business logic, such as loading the context from the global state store. Asynchronous non-blocking has two meanings: first, the listener's message retrieval operation itself is non-blocking and does not occupy the Worker's main computing thread; second, multiple Workers can consume different message partitions under the same topic in parallel, achieving horizontally scalable consumption capabilities. For example, in a decision queue configured with four Workers, each Worker independently subscribes to and processes a portion of the routing credentials. When a Worker enters an IO waiting state due to calling an external API, other Workers can still continue to consume new credentials without interference. This mechanism ensures that even if latency jitter occurs in a single processing stage, the overall pipeline throughput can still be maintained at a high level. Meanwhile, since the flow is always lightweight preset routing credentials rather than heavy state data, the network bandwidth consumption and memory overhead of the message middleware are compressed to an extremely low level, enabling the scheduling signaling of thousands of concurrent evaluation rounds to flow smoothly in a normal gigabit network environment, significantly reducing the cost threshold of infrastructure.

[0036] Step S30: Based on the task identifier of the preset routing credential in the flow, obtain the running context of the heterogeneous computing task from the global state storage area, and send the execution instruction of the current stage to the target execution environment according to the running context.

[0037] In practice, when a Worker in the asynchronous processing queue consumes a preset routing credential, it does not execute the task directly. Instead, it first uses the task identifier in the credential as an index key to retrieve and load the corresponding runtime context from the global state store (ctx_store). The global state store is a centralized in-memory dictionary or distributed cache responsible for maintaining the complete lifecycle state of all active tasks. Only after successfully obtaining the runtime context and completing the necessary dependency checks will the Worker issue specific execution instructions to the target execution environment, such as a remote large model API, a local sandbox container, or a decision service, based on the current stage configuration recorded in the context.

[0038] Step S40: In response to the completion of the execution instruction of the current stage, determine the asynchronous processing queue of the next stage from multiple asynchronous processing queues according to the task attribute information of the heterogeneous computing task, and asynchronously route the preset routing credential to the asynchronous processing queue of the next stage.

[0039] When a certain stage, such as the inference stage in the generation queue, is completed, the system does not mechanically pass the task to a fixed next queue. Instead, it rereads the resource configuration fields (such as step_resource) and isolation allocation mode (allocation_mode) from the task attribute information and dynamically decides where to go in the next stage based on this. For example, if the inference result indicates that code verification is required and the task is configured in task-level shared mode, the default routing credentials will be routed to the shared sandbox queue; if terminal interaction is required and the round-level exclusive mode is configured, the task will be routed to the exclusive sandbox queue; if only LLM scoring is required, the task will be routed to the decision queue; if the task has terminated or encountered an error, the task will be routed to the output queue.

[0040] This embodiment provides an asynchronous scheduling method for heterogeneous computing tasks. The method involves acquiring a heterogeneous computing task to be processed, parsing its task attribute information to obtain its resource requirement type, and allocating the task to an asynchronous processing queue on a target computing node in a multi-node cluster environment based on the resource requirement type. Within the asynchronous processing queue, a preset routing credential corresponding to the heterogeneous computing task is acquired asynchronously and transferred. The preset routing credential includes at least a task identifier. Based on the task identifier of the transferred preset routing credential, the running context of the heterogeneous computing task is retrieved from the global state storage area, and an execution instruction for the current stage is sent to the target execution environment according to the running context. Upon completion of the execution instruction for the current stage, the next asynchronous processing queue is determined from multiple asynchronous processing queues based on the task attribute information of the heterogeneous computing task, and the preset routing credential is asynchronously routed to the next asynchronous processing queue. By decoupling task scheduling from state storage, lightweight preset routing credentials are used to flow between queues, avoiding memory bloat and consistency risks caused by copying the complete context between queues. At the same time, the dynamic routing mechanism based on task attribute information enables tasks with different resource requirements to be automatically diverted to the corresponding dedicated queues, eliminating mutual blocking caused by heterogeneous resource sharing thread pools and improving the overall throughput and resource utilization of the system.

[0041] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S10 includes steps S101 to S106: Step S101: Obtain the heterogeneous computing task to be processed, parse the task attribute information of the heterogeneous computing task, and obtain the resource requirement type of the heterogeneous computing task. The resource requirement type includes at least one of graphics processor inference resources, central processing unit computing resources, exclusive sandbox resources, and shared sandbox resources.

[0042] In practice, the orchestrator receives evaluation requests from external sources via API interfaces or message subscription mechanisms. These requests constitute heterogeneous computing tasks to be processed, encapsulating all the metadata required to perform embodied intelligence evaluations. Unlike traditional tasks that contain only simple instructions, the heterogeneous computing tasks in this embodiment carry rich structured attributes to guide subsequent scheduling decisions.

[0043] It's important to note that the system extracts key fields from task attribute information and maps them to standard resource requirement types. Resource requirement types are abstract classifications of underlying physical computing resources, including but not limited to graphics processing unit (GPU) inference resources, central processing unit (CPU) computing resources, dedicated sandbox resources, and shared sandbox resources. For example, when the `step_resource` field in the task configuration is parsed to be "judge", the system determines that the task belongs to a GPU inference resource or remote API call requirement; when the `step_resource` field is parsed to be "sandbox" and `allocation_mode` is "per_episode", it is determined to be a dedicated sandbox resource requirement. This translates the business-level evaluation intent into resource semantics recognizable by the scheduling system, laying the foundation for subsequent accurate matching.

[0044] In addition to the resource requirement types mentioned above, other resource requirement types of queues can be added according to the same "subdivided queue + independent worker pool + dynamic routing" paradigm, such as GPU local inference queue, real robot hardware queue, and dedicated sensor simulation queue.

[0045] The granularity of resource requirement types can be expanded according to the actual hardware configuration of the cluster. For example, it can also include FPGA acceleration resources, dedicated AI chip resources, etc., as long as different physical execution environments can be distinguished.

[0046] Step S102: Obtain the real-time resource level of each computing node in the multi-node cluster corresponding to the resource demand type.

[0047] The system maintains a cluster node capability profile table, recording the hardware characteristics and resource tags of each computing node. When determining a target computing node, the system iterates through this profile table, filtering for nodes whose hardware capabilities match their resource requirements. For example, when the resource requirement is a graphics processing unit (GPU) inference resource, the system will lock onto nodes with GPU drivers, sufficient video memory capacity, and in a healthy state as candidates; while when the resource requirement is a central processing unit (CPU) computing resource, it prioritizes nodes with a higher number of CPU cores and sufficient memory bandwidth. This resource-attribute-based targeted selection mechanism achieves precise "task-node" matching, completely different from the blind scheduling strategies of random allocation or simple round-robin in traditional technologies.

[0048] Real-time resource availability refers not only to the total remaining hardware resources on a computing node, but also to the resource topology and physical affinity status. For example, for graphics processing unit (GPU) inference resources, the system not only records the number of remaining GPUs on a node, but also identifies whether these GPUs are in the same NVLink domain or NUMA node. This fine-grained resource availability mechanism allows subsequent resource allocation to be aware of the underlying hardware's physical constraints, avoiding communication bottlenecks caused by allocating distributed training tasks requiring high-speed interconnection to discrete GPUs across PCIe switches.

[0049] Step S103: Based on the real-time resource level, a greedy binning algorithm is used to select candidate computing nodes from the computing nodes that meet the resource requirements of the heterogeneous computing task.

[0050] The core idea of ​​the greedy binning algorithm is to prioritize fitting tasks into the smallest resource blocks that can meet their needs, or to attempt to fill resource blocks in descending order of a preset resource block size, in order to maximize the preservation of large, contiguous resource blocks for subsequent, larger-scale tasks. Unlike traditional polling or random allocation strategies, the greedy binning algorithm explicitly considers resource fragmentation. For example, on a computing node with 24 GPUs, the system's preset binning size sequence is [8, 4, 2]. When a heterogeneous computing task requiring 8 GPUs is received, the algorithm first checks whether the node has a complete 8-GPU resource block; if it does, it marks the node as a candidate computing node and locks the resource block; if there is no complete 8-GPU block but there are two 4-GPU blocks, and the task can be split (depending on the specific configuration), it attempts to combine and allocate them. Through this deterministic binning logic, the system can normalize fragmented resource requests, significantly reducing the probability of large tasks being unschedulable due to resource fragmentation. The greedy binning algorithm can also be replaced by optimal bin-packing (exact solution), weighted allocation based on job priority, or allocation strategy combined with topology-aware (NVLink / NUMA affinity). This embodiment does not impose any restrictions on this.

[0051] Step S104: Obtain the backlog of jobs to be run on each candidate computing node, and determine the candidate computing node whose backlog does not exceed the preset back pressure threshold as the target computing node.

[0052] The backlog of jobs reflects the current load saturation of compute nodes and the risk of task queuing delays. The preset backpressure threshold is a configurable safety boundary parameter used to prevent node overload or response timeouts caused by sudden surges in traffic. For example, if a candidate compute node has sufficient physical resources but already has 15 jobs in its job queue, and the preset backpressure threshold is set to 10, then that node will be temporarily excluded from the current scheduling selection. This mechanism constitutes the second line of defense in resource scheduling, adding runtime load considerations to physical resource matching.

[0053] Step S105: When the backlog of all candidates exceeds the preset back pressure threshold, trigger an independent orchestration instance to perform elastic scaling on the candidate computing nodes and determine the scaled-up candidate computing nodes as the target computing nodes.

[0054] When all candidate compute nodes within the cluster that meet the physical resource requirements are excluded due to backlog exceeding limits, it indicates that the current cluster's processing capacity has reached its bottleneck. At this point, the system will not directly reject tasks or allow them to wait indefinitely at the entry point. Instead, it will trigger an independent orchestration instance to perform elastic scaling. An independent orchestration instance is an asynchronous control component decoupled from the main scheduling loop, specifically responsible for requesting new compute resources from the underlying infrastructure or vertically scaling existing nodes. For example, when it detects that the job backlog on all GPU nodes exceeds the threshold, the independent orchestration instance will call the cloud platform's AutoScaling interface to request a new GPU compute node with the same specifications. Once the new node is ready and registered in the cluster, it will be used as the target compute node to accept the current task. Decoupling scheduling decisions and resource provisioning on the timeline ensures that the low-latency characteristics of the main scheduling loop are not affected by the time consumed by scaling operations. Simultaneously, scaling driven by backpressure signals achieves precise linkage between resource provisioning and actual load, avoiding the cost waste caused by reserving excessive redundant resources.

[0055] In one feasible implementation, to prevent high-priority tasks from not being scheduled in a timely manner, the system will also add a mechanism for automatic yielding to high-priority tasks before elastic expansion is triggered. Therefore, after step S105, the system further includes: The resource usage status of the target computing node is monitored through a separately running monitoring thread; If the resource occupancy status is detected to have reached a preset full load threshold and there is an external task request with a priority higher than the current priority to access the target node, the historical heterogeneous computing tasks submitted to the target computing node are automatically canceled to release the computing resources of the target computing node for external task scheduling.

[0056] It should be noted that the monitoring thread is the job monitor to be run. The thread is independent of the regular task scheduling main loop and continuously monitors the real-time resource usage of the target computing node (such as GPU utilization, video memory usage, CPU load, etc.) to determine whether the node is in a "full load" state.

[0057] The preset full-load threshold can be set in advance, for example, to 95%. If GPU utilization consistently exceeds 95%, it means the node has no available resources to accept new tasks. If an "external high-priority task" requests to join at this time, a yielding action is triggered. External tasks typically refer to tasks that are not part of the current regular scheduling queue but have higher business priority. Subsequent yielding actions are only triggered when the node is full and high-priority tasks cannot join. This avoids mistakenly killing running tasks when node resources are sufficient.

[0058] At this point, the scheduling system proactively intervenes, forcibly canceling historical heterogeneous computing tasks currently running on the current node. For example, it prioritizes canceling the lowest priority or shortest-running tasks to minimize business losses. The canceled tasks release their occupied GPU / CPU resources. The resource usage of the target computing node then decreases, and the system immediately allocates the released resources to waiting external high-priority tasks, allowing them to begin execution immediately. The PendingMonitor's strategy of canceling the oldest job can be replaced with canceling the job with the least progress, preemption based on job priority, or fair scheduling based on quotas.

[0059] In one feasible implementation, for example, the cluster configuration is queue_names=[rc-gpu-h100], split_sizes=[8], and main_loop_interval=300s. First round: AutoDeployLoop finds allocable_gpu=24 and no jobs to run, GpuAllocator.split(24)=[8,8,8], and submits 3 Fuyao jobs with 8 GPUs each in sequence. Each job runs an independent Orchestrator and shares the inference service endpoint. Backpressure takes effect: If there is an external high-priority training job in the queue JOB_PENDING in a certain round, AutoDeployLoop skips the new submission in the queue. After PendingMonitor identifies that it is not a job of this system and is not a protected account, it cancels the job submitted earliest in the queue by this system and cleans up the workspace, releasing 8 GPUs to high-priority tasks; in the next round, AutoDeployLoop can reallocate GPUs. Once a job is completed, `prune_completed_jobs()` removes it from the tracking list, freeing up the GPU for reallocation in the next round.

[0060] Step S106: On the target computing node, the heterogeneous computing task is assigned to an asynchronous processing queue corresponding to the resource requirement type.

[0061] After selecting a target computing node, the system further locates the specific asynchronous processing queue within that node based on resource requirements. Each computing node has multiple pre-defined independent queues categorized by resource requirements. For example, if the target node is selected for code execution evaluation, corresponding to exclusive sandbox resources, heterogeneous computing tasks will be directly submitted to the exclusive sandbox queue on that node to await consumption. If the task only requires mathematical formula derivation, corresponding to CPU computing resources, it will be allocated to the CPU computing queue. This node-level secondary precise routing ensures that tasks enter an execution channel perfectly suited to their resource consumption characteristics. This enables complete traffic isolation and concurrency decoupling between I / O-intensive and compute-intensive tasks within the physical node, providing the prerequisite for independent adjustment of concurrency parameters for each queue in the future.

[0062] In one feasible implementation, step S106 may include steps B11 to B14: Step B11: Determine the current supply capacity on the target computing node according to the resource demand type; Current supply capacity refers to the upper limit of the actual service capacity that a target computing node can provide for a specific type of resource demand. It is not a static hardware nominal value, but a dynamic indicator that combines hardware specifications, software configuration and external constraints.

[0063] The metrics for supply capacity differ depending on the type of resource. For example, when the resource requirement is a graphics processing unit (GPU) inference resource or a remote API call, the current supply capacity is typically determined by the rate limit of the external service or the maximum concurrency of the local network connection pool. If a large model inference interface is limited to 100 queries per second (QPS), then the supply capacity of that node is 100 QPS. When the resource requirement is a CPU computing resource, the supply capacity directly depends on the number of physical cores available to the node or the allocated vCPU quota. When the resource requirement is a dedicated sandbox resource, the supply capacity is limited by the total memory of the node, the quota of the container orchestration platform, or the preset maximum number of active containers.

[0064] Step B12: Match the preset resource-queue mapping relationship according to the resource requirement type, and match the asynchronous processing queue corresponding to the resource requirement type from multiple asynchronous processing queues. Different asynchronous processing queues correspond to different resource requirement types, and each asynchronous processing queue is independently configured with execution resources. Resource-queue mapping is a configuration strategy pre-defined in the orchestrator, which defines the binding relationship between various resource requirement types and specific asynchronous processing queues. For example, the generation queue is dedicated to serving remote API inference tasks, the decision queue serves scoring and decision tasks, the shared sandbox queue and the exclusive sandbox queue serve containerized tasks with different isolation levels, and the CPU computation queue serves local pure computation tasks.

[0065] The essence of this mapping relationship is to abstract heterogeneous execution environments into standardized queue interfaces. More importantly, each asynchronous processing queue is independently configured with execution resources. This means that each queue has its own dedicated pool of Workers, and the number, lifecycle, and runtime parameters of these Workers can be set independently, without resource sharing or lock contention between them. For example, the Workers in the generation queue focus on sending HTTP requests and waiting for responses, while the Workers in the CPU computation queue focus on numerical calculations. This physical isolation design completely eliminates the head-of-queue blocking effect caused by I / O-intensive tasks blocking computationally intensive tasks in traditional single shared thread pools, ensuring that various heterogeneous tasks can flow in parallel according to their own rhythm.

[0066] By configuring an independent worker pool for each queue, the number of workers is independently set through the configuration class `ConcurrencyConfig`. This independent configuration of the worker pool allows the system to independently adjust concurrency based on the actual supply capacity of each resource demand type: the number of `generation` workers matches the number of connections to the remote inference service, the number of `step_sandbox_episode` workers matches the number of available sandbox containers, and the number of `step_cpu` workers matches the number of CPU cores. There are no synchronization barriers between queues; after completing one step in an evaluation round, a worker can immediately return to the `generation` queue to wait for the next inference step, without waiting for other rounds in the same batch to complete. Each worker instance inherits from the `BaseWorker` abstract base class and executes a unified `run_forever()` asynchronous loop.

[0067] Step B13: Adjust the concurrent processing parameters of each asynchronous processing queue according to the current supply capacity; Concurrency parameters mainly refer to the number of active Workers in the asynchronous processing queue or the token bucket filling rate. This step establishes a dynamic mapping mechanism from supply capacity to concurrency parameters, enabling the queue's processing rate to accurately adapt to actual resource bottlenecks. For example, if the supply capacity of the remote API is determined to be 100 QPS, the number of concurrent Workers in the generation queue is set to 100 or slightly lower, thus ensuring throughput while avoiding server-side circuit breaking due to exceeding limits; if only 4 CPU cores are available locally, the number of concurrent Workers in the CPU computing queue is strictly limited to 4 to prevent a surge in context switching overhead caused by excessive thread creation; if the current node supports a maximum of 20 concurrent sandbox instances, the concurrency of the exclusive sandbox queue is set to 20.

[0068] For example, when a change in API rate limiting policy or resource preemption is detected, the system can automatically recalculate and hot-update concurrency parameters. Compared to the fixed-size thread pool solution commonly used in existing technologies, this embodiment achieves deep alignment between the scheduling system and the underlying heterogeneous resource supply characteristics through adaptive concurrency adjustment driven by supply capacity. This prevents resource contention and external penalties caused by excessive concurrency, and also avoids resource idleness caused by excessively low concurrency, thereby maintaining optimal throughput efficiency in complex and ever-changing cluster environments.

[0069] Step B14: Based on the adjusted concurrent processing parameters, allocate the heterogeneous computing tasks to the asynchronous processing queues corresponding to the resource requirement types.

[0070] After adjusting the concurrency parameters, the orchestrator delivers the heterogeneous computing tasks to be processed to the asynchronous processing queues corresponding to the resource requirement types. At this point, the queue has the processing capacity to match the current resource supply and can consume tasks at the expected rate. For example, when a task requiring GPU inference is assigned to the generation queue, because the number of workers in the queue has been precisely adjusted to within the API rate limiting threshold, the task can be processed smoothly without causing backlog or errors; when a code execution task is assigned to the exclusive sandbox queue, because the concurrency is controlled by the container instance limit, the system ensures that each scheduled task can immediately obtain an independent execution environment without waiting for the container to be released in the queue.

[0071] This embodiment derives the corresponding resource requirement type by parsing the attribute information of heterogeneous computing tasks. It can accurately match different heterogeneous resources such as GPU inference resources, CPU computing resources, dedicated sandbox resources, and shared sandbox resources, avoiding resource mismatch problems in traditional scheduling. This ensures that different types of computing resources are fully utilized, reducing resource idleness and waste, and maximizing the value of heterogeneous resources in a multi-node cluster. Using the real-time resource level of each computing node's corresponding resource requirement type as the filtering criterion, the current resource usage of nodes can be intuitively grasped. Combined with a greedy binning algorithm, candidate nodes that meet the resource requirements are quickly selected, ensuring both filtering efficiency and that the selected nodes have sufficient resources to handle new tasks, avoiding task execution lag due to resource estimation errors. By statistically analyzing the number of pending jobs on candidate computing nodes and comparing it with a preset backpressure threshold, target nodes can be filtered. Nodes with severe job backlogs can be proactively avoided, and new tasks can be allocated to computing nodes with reasonable loads, preventing a decline in overall task processing efficiency due to excessive load on a single node, resulting in a more balanced cluster load distribution. When the job backlog on all candidate nodes exceeds the back pressure threshold, an independent orchestration instance is automatically triggered to complete elastic scaling. This eliminates the need for manual pre-reservation of excessive redundant resources and allows for rapid replenishment of computing resources when the cluster load exceeds its current capacity, ensuring that new heterogeneous computing tasks are processed in a timely manner.

[0072] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S40 includes steps S401 to S402: Step S401: In response to the completion of the execution instruction of the current stage, obtain the resource attributes and isolation allocation mode configured in the task attribute information of the heterogeneous computing task.

[0073] When an asynchronous processing queue completes its current stage of processing (e.g., a large model inference returns a result), the orchestrator does not mechanically pass the pre-defined routing credentials to a fixed downstream queue. Instead, it rereads the task attribute information maintained by the heterogeneous computing task in the global state storage area. The resource attribute (step_resource) is an enumerated field that identifies the type of execution environment required for the next stage. Its values ​​include, but are not limited to, judge, sandbox, and local computing (CPU), representing the task's macro-level requirements for underlying computing power or services. The allocation mode (allocation_mode) is a fine-grained control parameter for specific resource attributes (especially sandboxes), used to define the lifecycle and reuse strategy of execution instances. Typical values ​​include task-level sharing (shared_task) and round-level exclusive access (per_episode).

[0074] It should be understood that these two fields together constitute a two-dimensional routing decision space: resource attributes determine "where to execute," while isolation allocation mode determines "at what level of security and efficiency to execute." This design, which explicitly configures routing decision factors, enables the scheduling system to flexibly adapt to the ever-changing business flows in embodied intelligent evaluation without having to hard-code complex jump logic.

[0075] Step S402: Determine the next-stage asynchronous processing queue from multiple asynchronous processing queues based on the resource attributes and the isolation allocation mode, and asynchronously route the preset routing credentials to the next-stage asynchronous processing queue.

[0076] Understandably, the system first performs primary routing based on resource attributes. If the resource attribute indicates the need for a non-containerized lightweight service, such as a judgment service or local computation, it directly maps to the corresponding dedicated queue. If the resource attribute indicates the need for an isolated execution environment, a secondary routing decision is triggered, further directing traffic precisely to sandbox queues with different lifecycle management strategies based on the isolation allocation mode. For example, in a typical hybrid evaluation scenario, after the generation queue completes inference, if the resource attribute of a task is read as "sandbox" and the isolation allocation mode is "shared_task", the system will asynchronously route the task's preset routing credentials to the shared sandbox queue, allowing it to reuse existing container instances to reduce startup overhead. Conversely, if the resource attribute is also "sandbox" but the isolation allocation mode is "per_episode", the preset routing credentials are routed to the exclusive sandbox queue to ensure that each evaluation round is executed in a completely new and clean environment, meeting high-security isolation requirements. As another example, if the resource attribute is "judge", it directly routes to the judgment queue to call the scoring service. Through this cascading judgment mechanism, this application achieves seamless coexistence and automatic traffic splitting of multiple heterogeneous business flows within a single runtime. Its core technical advantage lies in abstracting complex business flow logic into configurable metadata-driven processes. This not only eliminates the head-of-line blocking problem caused by resource mismatch in traditional fixed pipelines but also gives the system extremely strong scalability.

[0077] In one feasible implementation, step S402 may include steps C11 to C14: Step C11: If the resource attribute indicates that a determination service is required, determine the asynchronous processing queue for the next stage as the determination service queue, and route the preset routing credential to the determination service queue. Decision services typically refer to I / O-intensive tasks that rely on external large language model APIs or internal dedicated scoring models. Their core characteristics are that network communication time is significantly longer than local computation time, and they do not require isolated code execution environments. For example, in a mathematical reasoning evaluation task, after the generation queue completes the generation of the question's answer, the system reads the task configuration and finds the resource attribute is "judge." At this point, the orchestrator directly sends a pre-defined routing credential containing the task identifier to the decision queue. Upon receiving the credential, the worker in the decision queue loads the question and the model-generated answer from the global state store, constructs a scoring prompt, and calls the remote LLM scoring interface to obtain the score. Because the decision queue is independently configured with concurrency parameters, it can efficiently handle a large number of concurrent scoring requests without blocking other types of tasks while waiting for network responses.

[0078] Step C12: If the resource attribute indicates that computing services are needed, determine that the asynchronous processing queue for the next stage is the local computing queue, and route the preset routing credentials to the local computing queue.

[0079] Computation services refer to pure software computation tasks that rely entirely on local CPU resources, including but not limited to data preprocessing, format conversion, metric statistics, and result aggregation. These tasks are characterized by deterministic computational logic, no external I / O dependencies, and no need for security isolation. For example, in the data preparation phase before an evaluation task starts, or in the result aggregation phase after the evaluation, if the resource attribute is marked as "cpu," a pre-defined routing credential will be routed to the CPU computation queue. Workers in the CPU computation queue execute the corresponding computational logic directly in their local processes, without needing to create or enter any container environment. Compared to the container startup and environment initialization overhead of sandbox tasks, which can easily reach hundreds of milliseconds or even seconds, the task startup time of the local computation queue is only in the microsecond range, significantly improving the processing efficiency of lightweight tasks. Furthermore, because the CPU computation queue is physically isolated from the generation queue, decision queue, etc., even if a local computation task causes the CPU to be instantly fully loaded due to excessive algorithm complexity, it will not interfere with latency-sensitive inference requests or sandbox interaction instructions, ensuring the peaceful coexistence of heterogeneous loads on the same node.

[0080] Step C13: If the resource attribute indicates that the execution environment needs to be isolated and the isolation allocation mode is task-level sharing, determine the asynchronous processing queue for the next stage as the shared isolated execution environment queue, and asynchronously route the preset routing credentials to the shared isolated execution environment queue. Task-level sharing mode is primarily designed for business scenarios that are sensitive to startup latency and have no state conflicts or strong security isolation requirements between different evaluation rounds, such as code execution evaluation or algorithm verification tasks. In this mode, multiple evaluation rounds belonging to the same evaluation task or having the same configuration fingerprint (scope_key) are allowed to reuse the same sandbox container instance. When the orchestrator determines that the current task conforms to this mode, it sends the preset routing credentials to the shared sandbox queue, i.e., the shared isolated execution environment queue. At the underlying resource management level, the sandbox manager manages these shared instances by maintaining a shared entry lock containing asynchronous locks and reference counts. When the credentials for the first round arrive, the system creates the container and sets the reference count to 1; when credentials arrive in subsequent rounds, only the lock needs to be acquired and the reference count incremented to directly obtain the container handle, without having to go through the time-consuming image fetching and environment initialization process again. For example, in an evaluation task containing 100 Python script test cases, if a task-level sharing mode is adopted, the system may only need to create one container to complete all tests. Compared with the traditional method of creating a new container for each test case, this greatly improves the overall throughput of IO-intensive code evaluation.

[0081] Step C14: If the resource attribute indicates that an isolated execution environment is required and the isolation allocation mode is round-based exclusive, determine the asynchronous processing queue for the next stage as the exclusive isolated execution environment queue, and asynchronously route the preset routing credential to the exclusive isolated execution environment queue.

[0082] The round-based exclusive mode is primarily designed for business scenarios with strict requirements for environmental purity, potential malicious code risks, or the need to simulate independent sessions of real users. Typical examples include terminal interaction evaluation, operating system security testing, or multi-agent adversarial exercises. In this mode, each evaluation round must have a completely new, isolated execution environment that has not been used by any other round. When the orchestrator determines that the current task conforms to this mode, it sends the preset routing credentials to the exclusive sandbox queue, i.e., the exclusive isolated execution environment queue. At the underlying resource management level, the sandbox manager strictly controls the number of exclusive containers running simultaneously through activity-gated semaphores, and combines this with deployment-gated semaphores to suppress the impact of instantaneous creation requests on the underlying platform. Whenever a preset routing credential is consumed, the Worker first attempts to acquire an activity-gated semaphore slot. If successful, it then initiates a creation request to the container orchestration platform via deployment-gated semaphores at a rate limit. Once the container is ready, it returns a handle for the task to use. After the task ends, the container is immediately destroyed and the semaphore slot is released. For example, when evaluating Linux terminal operation capabilities, in order to prevent historical command records, environment variable modifications, or file system changes left over from the previous round from interfering with the scoring accuracy of subsequent rounds, the system forces the use of a round-level exclusive mode to ensure that each evaluation round is started in a pristine (pure) shell environment.

[0083] It is understood that, based on the above, this embodiment further provides a two-stage orphan recycling mechanism based on lease logs and atomic renaming to solve the problem of execution resource leakage caused by abnormal process termination. Therefore, after step S40, the method further includes steps S41 to S43: Step S41: During the execution of the heterogeneous computing task, a lease log is maintained in the global state storage area in an append-only manner. The lease log is used to record lease acquisition events and lease release events of the resources requested by the heterogeneous computing task. It should be noted that the lease log is the authoritative data source for recording lifecycle events of execution resources, and it is stored in the global state store or a separate persistent storage path. This embodiment maintains the log using an append-only approach, for example, using JSONL (JSON Lines) format, where each line records an independent JSON object. This avoids the lock contention and full rewrite overhead that occurs with frequent updates of traditional databases or structured files, greatly reducing the impact of IO load on the main scheduling chain.

[0084] Each event recorded in the log includes an event type (such as lease.acquire, lease.release.start, lease.release.end), a globally unique task identifier (ctx_id), an execution resource identifier (sandbox_id or job_name), and a timestamp accurate to milliseconds. Meanwhile, the Worker or sandbox manager responsible for managing execution resources internally runs a lightweight event acquisition mechanism. This mechanism appends a record to the lease log at preset time intervals (e.g., every 30 seconds), recording lease acquisition and lease release events corresponding to heterogeneous computing tasks.

[0085] Step S42: Periodically scan the lease log, and if an abnormal task is identified where a lease acquisition event exists but no corresponding lease release event is recorded, reclaim the execution resources occupied by the abnormal task; During the execution of heterogeneous computing tasks, lease logs can be periodically scanned to review all active task leases and identify abnormal tasks that have expired but have not yet recorded release events. These abnormal tasks typically correspond to heterogeneous computing tasks that terminated abnormally due to worker process crashes, network interruptions, or hardware failures. If the execution resources they occupy (such as sandbox containers, GPU memory, and temporary storage quotas) are not cleaned up in a timely manner, they will gradually accumulate and form resource leaks, ultimately slowing down the operating efficiency of the entire cluster.

[0086] Once an abnormal task is identified, the system immediately triggers a resource reclamation process. This involves calling the API of the underlying container orchestration platform (such as Kubernetes, Docker, or Fuyao) to forcibly terminate the corresponding container instance or job and appending a forced release record to the lease log to close the lifecycle. For example, if a code execution evaluation task becomes unresponsive to heartbeats due to CPU overload caused by a malicious script written by a user, the monitoring process will automatically destroy its associated sandbox container after 90 seconds, preventing the container from indefinitely occupying valuable GPU or CPU resources. This proactive runtime inspection mechanism ensures that even in the event of application-layer logic failure, underlying resources can still be released promptly through out-of-band management.

[0087] Step S43: Before the heterogeneous computing task is executed, perform an atomic renaming operation on the lease log to generate a snapshot file, identify historical tasks that did not end normally based on the snapshot file, and reclaim the execution resources occupied by the historical tasks.

[0088] In extreme cases such as process termination by the operating system, node power failure, or system crash, the runtime heartbeat maintenance and recycling logic has no chance to execute, leaving legacy execution resources as complete "orphans." To safely clean up these historical legacies when a new process starts, this embodiment introduces a snapshot isolation mechanism based on file system atomic operations. During the initialization phase of a new orchestrator instance and before creating any new execution resources, the system first checks for the existence of lease log files left over from the previous run. If they exist, the operating system's underlying rename system call is invoked to atomically rename the log file to a snapshot file with a specific suffix (such as .stale.jsonl). The "atomicity" here is crucial; it ensures that no new data is written to the old file during the renaming process, and no read operations access a file in a half-written state, thus completely eliminating the risk of data races between the old and new runtime cycles. Subsequently, the system parses the snapshot file, traverses all event records, and constructs a resource state machine: if an execution resource is found to have only a lease.acquire event but lacks a matching lease.release.end event, it is identified as a historical orphan task that did not terminate normally. The system then performs a forced cleanup operation on these orphaned tasks. For example, if an operations and maintenance personnel manually kill the evaluation process due to excessive cluster load, the system will automatically identify and clean up the five exclusive sandbox containers that were not released during the last run upon the next startup, without requiring manual intervention.

[0089] In another implementation, the append-only JSONL registry can be replaced with an embedded database (SQLite), a time-series database, or a distributed key-value store; orphan determination can be replaced by a timed scan with an event-driven lease timeout (TTL / leaserenerwal) mechanism.

[0090] In one feasible implementation, the step of "issuing the execution instruction for the current stage to the target execution environment according to the running context" in step S30 may include steps D11 to D14: Step D11: Based on the runtime context, initiate a request to the sandbox manager to obtain sandbox resources, and receive the sandbox resource instance returned by the sandbox manager; It should be noted that the sandbox manager is responsible for the unified management of the lifecycle of all isolated execution environment instances, including container creation, resource quota allocation, status monitoring and destruction. All tasks that require isolated execution environments must obtain resources through this standardized interface, avoiding coupling and compatibility issues caused by business logic directly connecting to the underlying container platform.

[0091] In the request, the caller needs to pass in metadata information such as task identifier, resource attributes, isolation allocation mode and configuration fingerprint. The sandbox manager will match the corresponding resource allocation strategy based on this information and return the sandbox resource instance handle that meets the requirements.

[0092] Step D12: Based on the isolation allocation mode corresponding to the heterogeneous computing task, use the corresponding handle protocol to perform lifecycle management on the sandbox resource instance; Understandably, after obtaining a sandbox resource instance, lifecycle management can be performed on the sandbox resource instance.

[0093] It should be noted that, since heterogeneous computing tasks have different requirements for isolation environments, after the scheduling system obtains a sandbox resource instance, it will use a handle protocol that matches the isolation allocation mode of the current heterogeneous computing task to manage the lifecycle of the sandbox resource instance.

[0094] In specific implementation, when the isolation allocation mode corresponding to the heterogeneous computing task is task-level sharing, the concurrent control of the sandbox resource instance is performed through asynchronous locks and reference counting mechanisms, and the sandbox resource instance is kept alive when the reference count of the sandbox resource instance is greater than zero; when the isolation allocation mode corresponding to the heterogeneous computing task is round-level exclusive, the concurrent control of the sandbox resource instance is performed through activity gating semaphores to limit the number of exclusive sandbox resource instances that are running at the same time.

[0095] It should be noted that if the isolation allocation mode is indicated as task-level sharing, the scheduling system uses asynchronous locks and reference counting mechanisms to control the concurrency of sandbox resource instances. As long as the reference count of the sandbox resource instance is greater than zero (i.e., there are still tasks using it), the system will keep the sandbox resource instance alive, avoiding the overhead caused by frequent creation and destruction.

[0096] If the isolation allocation mode is indicated as round-specific exclusive, the scheduling system uses activity gating semaphores to control the concurrency of sandbox resource instances, strictly limiting the number of exclusive sandbox resource instances running simultaneously. The system also releases the sandbox resource instance immediately after the task ends, ensuring that the next round of tasks uses a completely new and clean isolation environment. Through this dual-mode handle protocol, the system can maximize the reuse of underlying sandbox resources while ensuring secure isolation.

[0097] In addition to Fuyao, the container orchestration backend of the sandbox can be replaced with Kubernetes, Docker, Firecracker microvirtual machines, etc., while the acquire / release handle protocol remains unchanged; the isolation level can be extended from containers to a multi-layered hierarchy of function calls, microvirtual machines, and full virtual machines.

[0098] For example, the sandbox configuration for terminal operation evaluation is backend=fuyao, profile=execd_terminal, allocation_mode=per_episode, protocol=execd. The policy parser parse_suite_sandbox_policy performs strict combination checks (execd_terminal must be paired with per_episode + execd) and prohibits cluster-level resource fields (site / partition / project, etc., which are uniformly injected by the outer cpu_config) from appearing in the provider. When SandboxManager is initialized, it creates _deploy_gate = Semaphore(deploy_concurrency) and _active_gates_by_profile[execd_terminal] = Semaphore(max_active_sandboxes). When acquiring a container, first `await active_gate.acquire()` obtains the activity slot, then `FuyaoBaseProvider.acquire()` is called at a rate limit via `_deploy_gate` to create the container (create_timeout=600s, ready_timeout=300s, health check polling 10s). Upon successful creation, a handle_id (UUID) is assigned, `_HandleRecord` is recorded, and written to the `lease.acquire` event. When releasing the container, `provider.release()` (using `sandbox.kill()` + `close()`) is called, and the activity slot is returned in the `finally` block. If creation fails, the slot is returned in the `except` block to prevent semaphore leakage.

[0099] Step D13: During the process of obtaining the sandbox resource instance, concurrent rate limiting is performed on the requests of the sandbox manager to create sandbox resources by deploying gating semaphores in order to suppress deployment storms.

[0100] It's important to note that when a large number of heterogeneous computing tasks simultaneously initiate sandbox creation requests, the concentrated surge in creation requests within a short period can put immense pressure on the container orchestration platform's API services and node storage network, potentially even triggering a platform collapse. By setting deployment gating semaphores, the number of concurrent creation requests can be limited to a preset threshold, allowing container creation operations to be smoothly and orderly distributed to the underlying platform, preventing instantaneous request spikes from impacting the stability of the entire cluster.

[0101] Step D14: Use the sandbox resource instance as the target execution environment and send the execution instructions of the current stage to the target execution environment.

[0102] In the process of the scheduling system requesting sandbox resource instances from the sandbox manager, that is, in the process of the sandbox manager creating sandbox resources, in order to prevent a large number of concurrent requests from causing the underlying container runtime to overload, the system introduces a deployment gate semaphore mechanism.

[0103] The fixed concurrency limit of deploy_gate can be replaced with a token bucket, adaptive rate (dynamically adjusted based on platform response latency), or priority queue.

[0104] Specifically, the scheduling system uses this deployment gating semaphore to rate-limit the concurrency of requests from the sandbox manager to create sandbox resources. When the number of creation requests reaches the upper limit of the deployment gating semaphore, subsequent creation requests will be suspended until a sandbox resource instance is created and releases the semaphore permission. This mechanism effectively suppresses deployment storms, smooths the creation rate of underlying resources, and thus ensures the stable operation of the entire heterogeneous computing scheduling system in high-concurrency scenarios.

[0105] Through the design of the two routing branches described above, the dynamic coexistence and automatic switching of the "efficiency-first" and "security-first" strategies are achieved within a single scheduling framework. Compared to the binary opposition of existing technologies, which either result in security risks due to full sharing or wasteful resource consumption due to full monopolization, this application allows users to flexibly specify isolation levels at the task configuration granularity. This enables the same system to efficiently handle large-scale code regression testing while securely executing high-risk interactive penetration testing.

[0106] like Figure 4 As shown, Figure 4 This is a schematic diagram of the architecture of an asynchronous scheduling system for heterogeneous computing tasks. The system consists of three main components: an orchestrator (10), a sandbox manager (20), and an autodeploy loop (30). After receiving a task, the orchestrator (10) first parses its task attribute information. This information not only contains the basic metadata of the task, but also clearly indicates the type of resource requirements of the task (such as GPU inference, shared containers, exclusive containers, or CPU computing).

[0107] In its implementation, orchestrator 10 acts as the core hub for task scheduling, internally maintaining a heterogeneous queue topology based on resource type. Orchestrator 10 first allocates tasks to corresponding asynchronous processing queues (such as GPU inference queues, shared sandbox queues, or exclusive sandbox queues) based on the parsed resource requirement type, and then starts the corresponding worker pool for concurrent processing. During processing, orchestrator 10 does not directly create containers, but instead uses lightweight envelopes to transfer task states between queues, storing the complete context in the global state area. When a task reaches a stage requiring a sandbox environment, orchestrator 10 initiates a resource acquisition request to sandbox manager 20. Sandbox manager 20 is responsible for uniformly managing the lifecycle of all sandbox containers, its core function being the execution of a dual-mode handle protocol. Upon receiving a request from orchestrator 10, sandbox manager 20 first uses a deployment gate semaphore for concurrency rate limiting to suppress deployment storms caused by a sudden surge in creation requests. Subsequently, differentiated processing is performed based on the isolation allocation mode in the task attributes: In task-level shared mode, the sandbox manager 20 uses asynchronous locks and reference counting mechanisms to ensure that tasks with the same configuration share the same sandbox instance, releasing resources only when the reference count reaches zero; in round-level exclusive mode, an active gate semaphore controls the number of concurrently running exclusive sandboxes, and the container is destroyed immediately after the task ends to release the quota. After completing resource allocation or release, the sandbox manager 20 returns handle information to the orchestrator 10 so that it can issue execution instructions. In specific implementation, the orchestrator 10 acts as the core hub for task scheduling, internally maintaining a heterogeneous queue topology structure based on resource type. The orchestrator 10 first allocates tasks to the corresponding asynchronous processing queues (such as GPU inference queues, shared sandbox queues, or exclusive sandbox queues) according to the parsed resource requirement type, and starts the corresponding worker pool for concurrent processing. During processing, orchestrator 10 does not directly create containers, but instead uses lightweight envelopes to transfer task states between queues and stores the complete context in the global state area. When a task flows to a stage that requires a sandbox environment, orchestrator 10 sends a resource acquisition request to sandbox manager 20.

[0108] The Sandbox Manager 20 is responsible for the unified management of the lifecycle of all sandbox containers. Its core function is to execute a dual-mode handle protocol. Upon receiving a request from the Orchestrator 10, the Sandbox Manager 20 first uses a Deploy Gate Semaphore for concurrency limiting to suppress deployment storms caused by a surge in instantaneous creation requests. Subsequently, it differentiates processing based on the isolation allocation mode in the task attributes: in task-level shared mode, the Sandbox Manager 20 employs asynchronous locks and reference counting to ensure that tasks with the same configuration share the same sandbox instance, releasing resources only when the reference count reaches zero; in round-level exclusive mode, it controls the number of concurrently running exclusive sandboxes using an Active Gate Semaphore, and immediately destroys the container after the task ends to release the quota. After completing resource allocation or release, the Sandbox Manager 20 returns handle information to the Orchestrator 10 so that it can issue execution instructions.

[0109] The auto-deployment loop 30 runs in a separate monitoring thread, responsible for elastic scaling and resource coordination in a multi-node cluster environment. The auto-deployment loop 30 periodically monitors the resource levels of each compute node in the cluster and the backlog of jobs waiting to run. When it detects that a node is fully loaded and there are external high-priority task requests, the auto-deployment loop 30 triggers an automatic yielding mechanism, instructing the orchestrator 10 to cancel lower-priority historical tasks on the current node, thereby freeing up computing resources for external high-priority tasks. Furthermore, if the overall cluster resources are insufficient, the auto-deployment loop 30 will also trigger elastic scaling of nodes based on a greedy binning algorithm and synchronize the expanded node information to the orchestrator 10, enabling it to schedule new tasks onto the newly added computing resources.

[0110] For example, a certain embodied intelligence assessment task simultaneously includes three suites: Suite A, knowledge question answering (step_resource=judge), Suite B, code execution (sandbox / shared_task), and Suite C, terminal operations (sandbox / per_episode). The concurrency configuration is as follows: concurrency: reset_light: 2generation: 4step_sandbox_shared: 1; reset_sandbox: 1step_judge: 2step_sandbox_episode: 3; step_cpu: 1; The orchestrator creates nine asynchronous queues and corresponding worker pools in `setup()`, instantiates an environment object for each suite, and binds runtime services (including the inference client). `_enqueue_initial()` loads the dataset, creates an `EpisodeContext` for each sample, and stores it in `ctx_store`, with `ctx_id` in the format "{suite_id}:{epoch}:{episode_id}". Samples from suite A are added to the `reset_light` queue, while samples from suites B and C are added to the `reset_sandbox` queue. Taking one cycle of suite B as an example: after a worker in `reset_sandbox` resets, it is routed to `generation` → after inference, the worker in `generation` is routed to `step_sandbox_shared` according to `shared_task` → code is executed in the shared sandbox; if it doesn't terminate, it returns to `generation`, and this cycle continues until terminated or `max_steps` is reached → the sample enters the `output` queue. The `_monitor_output()` function continuously consumes the output queue, converts each completed round into an `EvalResultV2` and writes it to the per-suite's `results.jsonl`, then pops the context from `ctx_store` to release memory.

[0111] In response to the completion of the execution instruction of the current stage, this embodiment obtains the resource attributes and isolation allocation mode configured in the task attribute information of the heterogeneous computing task; based on the resource attributes and / or the isolation allocation mode, it determines the asynchronous processing queue for the next stage from multiple asynchronous processing queues, and asynchronously routes the preset routing credentials to the asynchronous processing queue for the next stage. Through resource attributes and isolation allocation modes, differentiated routing of different types of computing tasks within the same scheduling framework is achieved. This avoids mutual blocking between tasks with different resource requirements and allows for matching corresponding sandbox management strategies for different security requirements, significantly improving the utilization rate and scheduling flexibility of heterogeneous cluster resources. It enables the same scheduling framework to flexibly adapt to various heterogeneous evaluation processes such as knowledge question answering, code execution, and terminal interaction. Furthermore, it uses a unified acquire-release handle protocol to manage the lifecycle of sandbox containers in both modes and suppresses deployment storms through hierarchical semaphores, ensuring that even if hundreds of rounds of simultaneous exclusive sandbox access are required, it will not impact the container orchestration platform.

[0112] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the asynchronous scheduling method for heterogeneous computing tasks in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0113] This application also provides an asynchronous scheduling device for heterogeneous computing tasks; please refer to [reference needed]. Figure 5 The asynchronous scheduling device for the heterogeneous computing tasks includes: The acquisition module 100 is used to acquire heterogeneous computing tasks to be processed, parse the task attribute information of the heterogeneous computing tasks, obtain the resource requirement type of the heterogeneous computing tasks, and allocate the heterogeneous computing tasks to the asynchronous processing queues on the target computing nodes in a multi-node cluster environment according to the resource requirement type.

[0114] The generation module 200 is used to obtain a preset routing credential corresponding to the heterogeneous computing task in the asynchronous processing queue in an asynchronous waiting manner, and to transfer the preset routing credential, wherein the preset routing credential includes at least a task identifier.

[0115] The acquisition module 100 is further configured to acquire the running context of the heterogeneous computing task from the global state storage area based on the task identifier of the preset routing credential in the flow, and send the execution instruction of the current stage to the target execution environment according to the running context.

[0116] The determination module 300 is used to respond to the completion of the execution instruction of the current stage, determine the asynchronous processing queue of the next stage from multiple asynchronous processing queues according to the task attribute information of the heterogeneous computing task, and asynchronously route the preset routing credential to the asynchronous processing queue of the next stage.

[0117] The asynchronous scheduling device for heterogeneous computing tasks provided in this application, employing the asynchronous scheduling method for heterogeneous computing tasks in the above embodiments, can solve the technical problems of low scheduling efficiency, chaotic resource lifecycle management, and poor multi-node scalability in existing solutions. Compared with the prior art, the beneficial effects of the asynchronous scheduling device for heterogeneous computing tasks provided in this application are the same as those of the asynchronous scheduling method for heterogeneous computing tasks provided in the above embodiments, and other technical features in the asynchronous scheduling device for heterogeneous computing tasks are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0118] In one embodiment, the acquisition module 100 is further configured to acquire the heterogeneous computing task to be processed, parse the task attribute information of the heterogeneous computing task, and obtain the resource requirement type of the heterogeneous computing task. The resource requirement type includes at least one of graphics processor inference resources, central processing unit computing resources, exclusive sandbox resources, and shared sandbox resources. Obtain the real-time resource level of each computing node in the multi-node cluster corresponding to the resource demand type; Based on the real-time resource level, a greedy binning algorithm is used to select candidate computing nodes from the computing nodes that meet the resource requirements of the heterogeneous computing task. Obtain the backlog of jobs to be run on each candidate computing node, and determine the candidate computing node whose backlog does not exceed a preset back pressure threshold as the target computing node; If the backlog exceeds the preset back pressure threshold, an independent orchestration instance is triggered to perform elastic scaling on the candidate compute nodes, and the scaled-up candidate compute nodes are determined as the target compute nodes. On the target computing node, the heterogeneous computing tasks are assigned to asynchronous processing queues corresponding to the resource requirement types.

[0119] In one embodiment, the acquisition module 100 is further configured to monitor the resource occupancy status of the target computing node through an independently running monitoring thread; If the resource occupancy status is detected to have reached a preset full load threshold and there is an external task request with a priority higher than the current priority to access the target node, the historical heterogeneous computing tasks submitted to the target computing node are automatically canceled to release the computing resources of the target computing node for external task scheduling.

[0120] In one embodiment, the determining module 300 is further configured to, in response to the completion of the execution instruction of the current stage, obtain the resource attributes and isolation allocation mode configured in the task attribute information of the heterogeneous computing task; Based on the resource attributes and the isolation allocation mode, the next asynchronous processing queue is determined from multiple asynchronous processing queues, and the preset routing credentials are asynchronously routed to the next asynchronous processing queue.

[0121] In one embodiment, the determining module 300 is further configured to, when the resource attribute indicates that a determination service is required, determine the asynchronous processing queue of the next stage as the determination service queue, and route the preset routing credential to the determination service queue. If the resource attribute indicates that computing services are required, the asynchronous processing queue for the next stage is determined to be the local computing queue, and the preset routing credentials are routed to the local computing queue. When the resource attribute indicates that an isolated execution environment is required and the isolation allocation mode is task-level sharing, the asynchronous processing queue for the next stage is determined to be the shared isolated execution environment queue, and the preset routing credentials are asynchronously routed to the shared isolated execution environment queue. If the resource attribute indicates that an isolated execution environment is required and the isolation allocation mode is round-based exclusive, the asynchronous processing queue for the next stage is determined to be the exclusive isolated execution environment queue, and the preset routing credential is asynchronously routed to the exclusive isolated execution environment queue.

[0122] In one embodiment, the apparatus further includes a recycling module, which is used to maintain a lease log in a global state storage area in an append-only manner during the execution of the heterogeneous computing task. The lease log is used to record lease acquisition events and lease release events of the resources requested by the heterogeneous computing task. The lease log is periodically scanned, and if an abnormal task is identified that has a lease acquisition event but no corresponding lease release event is recorded, the execution resources occupied by the abnormal task are reclaimed. Before the heterogeneous computing task is executed, an atomic renaming operation is performed on the lease log to generate a snapshot file. Based on the snapshot file, historical tasks that did not end normally are identified, and the execution resources occupied by the historical tasks are reclaimed.

[0123] In one embodiment, the determining module 300 is further configured to initiate a request to the sandbox manager to obtain sandbox resources based on the running context, and receive a sandbox resource instance returned by the sandbox manager; Based on the isolation allocation mode corresponding to the heterogeneous computing task, the corresponding handle protocol is used to manage the lifecycle of the sandbox resource instance; During the process of acquiring the sandbox resource instance, a gating semaphore is deployed to perform concurrent rate limiting on the requests of the sandbox manager to create sandbox resources, in order to suppress deployment storms; The sandbox resource instance is used as the target execution environment, and the execution instructions for the current stage are sent to the target execution environment.

[0124] In one embodiment, the determining module 300 is further configured to perform concurrent control on the sandbox resource instance through asynchronous locks and reference counting mechanisms when the isolation allocation mode corresponding to the heterogeneous computing task is task-level sharing, and maintain the live state of the sandbox resource instance when the reference count of the sandbox resource instance is greater than zero. When the isolation allocation mode corresponding to the heterogeneous computing task is round-based exclusive, the sandbox resource instances are concurrently controlled by an activity gating semaphore to limit the number of exclusive sandbox resource instances that are running at the same time.

[0125] This application provides an asynchronous scheduling device for heterogeneous computing tasks. The asynchronous scheduling device for heterogeneous computing tasks includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the asynchronous scheduling method for heterogeneous computing tasks in the above embodiment 1.

[0126] The following is for reference. Figure 6 This document illustrates a schematic diagram of an asynchronous scheduling device suitable for implementing heterogeneous computing tasks in the embodiments of this application. The asynchronous scheduling device for heterogeneous computing tasks in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), tablets, PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The asynchronous scheduling device for heterogeneous computing tasks shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0127] like Figure 6As shown, the asynchronous scheduling device for heterogeneous computing tasks may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the asynchronous scheduling device for heterogeneous computing tasks. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the asynchronous scheduling device for heterogeneous computing tasks to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows an asynchronous scheduling device for heterogeneous computing tasks with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0128] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0129] The asynchronous scheduling device for heterogeneous computing tasks provided in this application, employing the asynchronous scheduling method for heterogeneous computing tasks in the above embodiments, can solve the technical problems of low scheduling efficiency, chaotic resource lifecycle management, and poor multi-node scalability in existing solutions. Compared with the prior art, the beneficial effects of the asynchronous scheduling device for heterogeneous computing tasks provided in this application are the same as those of the asynchronous scheduling method for heterogeneous computing tasks provided in the above embodiments, and other technical features in this asynchronous scheduling device for heterogeneous computing tasks are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0130] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0131] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0132] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the asynchronous scheduling method for heterogeneous computing tasks in the above embodiments.

[0133] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0134] The aforementioned computer-readable storage medium may be included in an asynchronous scheduling device for heterogeneous computing tasks; or it may exist independently and not be assembled into an asynchronous scheduling device for heterogeneous computing tasks.

[0135] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by an asynchronous scheduling device for heterogeneous computing tasks, the asynchronous scheduling device for heterogeneous computing tasks performs the following actions: It acquires a heterogeneous computing task to be processed, parses the task attribute information of the heterogeneous computing task to obtain the resource requirement type of the heterogeneous computing task, and allocates the heterogeneous computing task to an asynchronous processing queue on a target computing node in a multi-node cluster environment according to the resource requirement type; in the asynchronous processing queue, it acquires a preset routing credential corresponding to the heterogeneous computing task in an asynchronous waiting manner, and circulates the preset routing credential, which includes at least a task identifier; based on the task identifier of the circulating preset routing credential, it acquires the running context of the heterogeneous computing task from a global state storage area, and sends the execution instruction for the current stage to the target execution environment according to the running context; in response to the completion of the execution instruction for the current stage, it determines the asynchronous processing queue for the next stage from multiple asynchronous processing queues according to the task attribute information of the heterogeneous computing task, and asynchronously routes the preset routing credential to the asynchronous processing queue for the next stage.

[0136] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0139] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the asynchronous scheduling method for the above-described heterogeneous computing tasks. This solves the technical problems of low heterogeneous load scheduling efficiency, chaotic resource lifecycle management, and poor multi-node scalability in existing solutions. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the asynchronous scheduling method for heterogeneous computing tasks provided in the above embodiments, and will not be elaborated upon here.

[0140] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the asynchronous scheduling method for heterogeneous computing tasks as described above.

[0141] The computer program product provided in this application can solve the technical problems of low efficiency in heterogeneous evaluation load scheduling, chaotic resource lifecycle management, and poor multi-node scalability in existing solutions. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the asynchronous scheduling method for heterogeneous computing tasks provided in the above embodiments, and will not be repeated here.

[0142] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An asynchronous scheduling method for heterogeneous computing tasks, characterized in that, The asynchronous scheduling method for the heterogeneous computing tasks includes: Obtain the heterogeneous computing task to be processed, parse the task attribute information of the heterogeneous computing task to obtain the resource requirement type of the heterogeneous computing task, and allocate the heterogeneous computing task to the asynchronous processing queue on the target computing node in the multi-node cluster environment according to the resource requirement type. In the asynchronous processing queue, a preset routing credential corresponding to the heterogeneous computing task is obtained in an asynchronous waiting manner, and the preset routing credential is transferred. The preset routing credential includes at least a task identifier. Based on the task identifier of the preset routing credential in the flow, the execution context of the heterogeneous computing task is obtained from the global state storage area, and the execution instruction of the current stage is sent to the target execution environment according to the execution context. In response to the completion of the execution instruction of the current stage, the asynchronous processing queue for the next stage is determined from multiple asynchronous processing queues based on the task attribute information of the heterogeneous computing task, and the preset routing credential is asynchronously routed to the asynchronous processing queue of the next stage.

2. The method as described in claim 1, characterized in that, The steps of acquiring the heterogeneous computing task to be processed, parsing the task attribute information of the heterogeneous computing task to obtain the resource requirement type of the heterogeneous computing task, and allocating the heterogeneous computing task to the asynchronous processing queue on the target computing node in a multi-node cluster environment according to the resource requirement type include: Obtain the heterogeneous computing task to be processed, parse the task attribute information of the heterogeneous computing task, and obtain the resource requirement type of the heterogeneous computing task. The resource requirement type includes at least one of graphics processor inference resources, central processing unit computing resources, exclusive sandbox resources, and shared sandbox resources. Obtain the real-time resource level of each computing node in the multi-node cluster corresponding to the resource demand type; Based on the real-time resource level, a greedy binning algorithm is used to select candidate computing nodes from the computing nodes that meet the resource requirements of the heterogeneous computing task. Obtain the backlog of jobs to be run on each candidate computing node, and determine the candidate computing node whose backlog does not exceed a preset back pressure threshold as the target computing node; If the backlog exceeds the preset back pressure threshold, an independent orchestration instance is triggered to perform elastic scaling on the candidate compute nodes, and the scaled-up candidate compute nodes are determined as the target compute nodes. On the target computing node, the heterogeneous computing tasks are assigned to asynchronous processing queues corresponding to the resource requirement types.

3. The method as described in claim 2, characterized in that, After the step of triggering an independent orchestration instance to perform elastic scaling on the candidate compute nodes when the backlog number exceeds the preset back pressure threshold, and determining the scaled-up candidate compute nodes as the target compute nodes, the method further includes: The resource usage status of the target computing node is monitored through a separately running monitoring thread; If the resource occupancy status is detected to have reached a preset full load threshold and there is an external task request with a priority higher than the current priority to access the target node, the historical heterogeneous computing tasks submitted to the target computing node are automatically canceled to release the computing resources of the target computing node for external task scheduling.

4. The method as described in claim 1, characterized in that, The step of responding to the completion of the execution instruction of the current stage, determining the asynchronous processing queue for the next stage from multiple asynchronous processing queues based on the task attribute information of the heterogeneous computing task, and asynchronously routing the preset routing credential to the asynchronous processing queue for the next stage includes: In response to the completion of the execution instruction of the current stage, the resource attributes and isolation allocation mode configured in the task attribute information of the heterogeneous computing task are obtained; Based on the resource attributes and the isolation allocation mode, the next asynchronous processing queue is determined from multiple asynchronous processing queues, and the preset routing credentials are asynchronously routed to the next asynchronous processing queue.

5. The method as described in claim 4, characterized in that, The step of determining the next-stage asynchronous processing queue from multiple asynchronous processing queues based on the resource attributes and the isolation allocation mode, and asynchronously routing the preset routing credentials to the next-stage asynchronous processing queue includes: If the resource attribute indicates that a decision service is required, the asynchronous processing queue for the next stage is determined as the decision service queue, and the preset routing credentials are routed to the decision service queue. If the resource attribute indicates that computing services are required, the asynchronous processing queue for the next stage is determined to be the local computing queue, and the preset routing credentials are routed to the local computing queue. When the resource attribute indicates that an isolated execution environment is required and the isolation allocation mode is task-level sharing, the asynchronous processing queue for the next stage is determined to be the shared isolated execution environment queue, and the preset routing credentials are asynchronously routed to the shared isolated execution environment queue. If the resource attribute indicates that an isolated execution environment is required and the isolation allocation mode is round-based exclusive, the asynchronous processing queue for the next stage is determined to be the exclusive isolated execution environment queue, and the preset routing credential is asynchronously routed to the exclusive isolated execution environment queue.

6. The method as described in claim 5, characterized in that, The method further includes: During the execution of the heterogeneous computing task, a lease log is maintained in the global state storage area in an append-only manner. The lease log is used to record lease acquisition events and lease release events of the resources requested by the heterogeneous computing task. The lease log is periodically scanned, and if an abnormal task is identified that has a lease acquisition event but no corresponding lease release event is recorded, the execution resources occupied by the abnormal task are reclaimed. Before the heterogeneous computing task is executed, an atomic renaming operation is performed on the lease log to generate a snapshot file. Based on the snapshot file, historical tasks that did not end normally are identified, and the execution resources occupied by the historical tasks are reclaimed.

7. The method as described in claim 5, characterized in that, The steps of issuing execution instructions for the current stage to the target execution environment based on the runtime context include: Based on the runtime context, a request to obtain sandbox resources is initiated to the sandbox manager, and the sandbox resource instance returned by the sandbox manager is received. Based on the isolation allocation mode corresponding to the heterogeneous computing task, the corresponding handle protocol is used to manage the lifecycle of the sandbox resource instance; During the process of acquiring the sandbox resource instance, a gating semaphore is deployed to perform concurrent rate limiting on the requests of the sandbox manager to create sandbox resources, in order to suppress deployment storms; The sandbox resource instance is used as the target execution environment, and the execution instructions for the current stage are sent to the target execution environment.

8. The method as described in claim 7, characterized in that, The step of managing the lifecycle of the sandbox resource instance using the corresponding handle protocol based on the isolation allocation mode corresponding to the heterogeneous computing task includes: When the isolation allocation mode corresponding to the heterogeneous computing task is task-level sharing, the concurrent control of the sandbox resource instance is performed through asynchronous locks and reference counting mechanism, and the sandbox resource instance is kept alive when the reference count of the sandbox resource instance is greater than zero. When the isolation allocation mode corresponding to the heterogeneous computing task is round-based exclusive, the sandbox resource instances are concurrently controlled by an activity gating semaphore to limit the number of exclusive sandbox resource instances that are running at the same time.

9. An asynchronous scheduling device for heterogeneous computing tasks, characterized in that, The device includes: The acquisition module is used to acquire heterogeneous computing tasks to be processed, parse the task attribute information of the heterogeneous computing tasks, obtain the resource requirement type of the heterogeneous computing tasks, and allocate the heterogeneous computing tasks to the asynchronous processing queues on the target computing nodes in a multi-node cluster environment according to the resource requirement type. A generation module is used to obtain a preset routing credential corresponding to the heterogeneous computing task in the asynchronous processing queue in an asynchronous waiting manner, and to transfer the preset routing credential, wherein the preset routing credential includes at least a task identifier; The acquisition module is further configured to acquire the running context of the heterogeneous computing task from the global state storage area based on the task identifier of the preset routing credential in the flow, and send the execution instruction of the current stage to the target execution environment according to the running context; The determination module is used to respond to the completion of the execution instruction of the current stage, determine the asynchronous processing queue of the next stage from multiple asynchronous processing queues according to the task attribute information of the heterogeneous computing task, and asynchronously route the preset routing credential to the asynchronous processing queue of the next stage.

10. An asynchronous scheduling device for heterogeneous computing tasks, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the asynchronous scheduling method for heterogeneous computing tasks as described in any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the asynchronous scheduling method for heterogeneous computing tasks as described in any one of claims 1 to 8.

Images