Distributed artificial intelligence inference task scheduling system and method facing heterogeneous computing power

Abstract

This invention relates to the fields of artificial intelligence and distributed computing technology, and discloses a distributed AI inference task scheduling system and method for heterogeneous computing power, solving problems such as coupling between scheduling and inference, lack of type awareness, and task suspension due to node failure. The system includes a scheduling service module, a resource-aware computing execution module, and a distributed caching module. The scheduling service module atomically writes task identifiers into independent sub-task queues according to task type; when the concurrent task count and memory availability are both sufficient, the computing node initiates a blocking pop request with priority to the queue corresponding to the resident model type, reusing or switching the model inference pipeline as needed; asynchronous heartbeats maintain the online status of nodes, and the scheduling node identifies faulty nodes based on heartbeat timeouts and performs rollback rescue and limited retries for suspended tasks. This invention achieves separation of scheduling and computing, reduces model switching overhead, and improves the inference throughput efficiency and reliability of heterogeneous computing power clusters.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and distributed computing technology, specifically to a distributed artificial intelligence inference task scheduling system and method for heterogeneous computing power. Background Technology

[0002] With the widespread application of generative artificial intelligence and large-scale visual reasoning models, the demand for diverse hardware computing power in AI inference services is increasing. In practical deployment scenarios, computing nodes are often composed of a mix of graphics processors of different models and memory specifications, forming heterogeneous computing power clusters. Existing technologies, represented by distributed asynchronous task frameworks based on Redis message queues (such as Celery and RQ), achieve distributed task scheduling by uniformly queuing tasks and having worker nodes compete for them, and have been widely used in AI inference scenarios. However, when the above-mentioned existing technical solutions are applied to heterogeneous computing power AI inference scenarios, the following main technical problems exist:

[0003] First, the scheduling node is coupled with the inference load. Existing solutions deploy the task scheduling logic and the AI ​​model inference logic on the same node, requiring the scheduling node to be configured with an expensive graphics processor; when the inference task consumes a large amount of computing resources, the scheduling response performance degrades, causing task distribution delays.

[0004] Second, the task queue lacks type awareness. Existing solutions use a unified global task queue. When computing nodes retrieve tasks from the queue, they cannot perceive the matching between the task type and their already resident models, leading to frequent and unnecessary model switching. For example, if two types of tasks, face effect processing (referring to performing technical transformations on the face region in an image within the legally authorized scope of the data subject, such as generating filter effects) and video rendering, occur alternately, if a node retrieves a non-resident task, it must release the GPU memory and load a new model each time. The model loading time accounts for a large proportion of the total task time, severely reducing the system throughput efficiency.

[0005] Third, the resource status of computing nodes lacks dual gating for task acquisition. Existing solutions only consider the number of concurrent tasks in the task acquisition mechanism of computing nodes, without simultaneously incorporating the detection of the GPU's available memory. In scenarios where multiple types of tasks are executed together, insufficient GPU memory may lead to task execution failures or system crashes.

[0006] Fourth, the node failure detection and task recovery mechanisms are inadequate. After a computing node fails, the tasks being executed may be suspended for an extended period, affecting system reliability, and there is a lack of effective restrictions on indefinite retries of abnormal tasks.

[0007] Fifth, task status management is rudimentary. Existing solutions use a uniform storage strategy for tasks at different lifecycle stages, resulting in low storage resource utilization or premature loss of historical data.

[0008] Sixth, there is an additional startup delay for the first inference after model switching. After the computing node completes the model switching and loads the new model weights, the inference engine needs to perform just-in-time (JIT) compilation optimization of the computation graph during the first formal inference, resulting in the first inference taking significantly longer than subsequent inferences; existing solutions lack a targeted preprocessing mechanism for this, affecting task response efficiency.

[0009] Therefore, a distributed artificial intelligence inference task scheduling system is needed that can sense the type of computing node residing in the model, implement dual resource gating, and achieve highly reliable task recovery.

[0010] The various models used in the artificial intelligence inference tasks involved in this invention (including but not limited to models used for tasks such as face effect processing, video rendering, and image generation) all belong to the well-known model architectures in this technical field. This invention does not limit the network structure, model parameters, or training methods of the above models. The innovation of this invention lies in the heterogeneous computing power distributed scheduling system and method for the above model inference tasks.

[0011] The task input data (including images, videos, and other data that may contain personal information) mentioned in this specification must be collected and processed with the explicit informed consent of the relevant data subject and in accordance with the provisions of the Personal Information Protection Law of the People's Republic of China, the Data Security Law of the People's Republic of China, and other relevant laws and regulations. The processed data is used only for the legitimate artificial intelligence reasoning and computing purposes described in this invention. Summary of the Invention

[0012] The fundamental technical problem this invention aims to solve is: how to achieve task-type-based segmented queue distribution, competitive acquisition with dual resource gating, priority routing guided by resident model type, and node fault detection and limited rollback rescue of suspended tasks based on a heartbeat mechanism, without configuring a graphics processor on the scheduling service node, thereby improving the inference throughput efficiency and system reliability of heterogeneous computing power clusters. This invention also addresses the following preferred technical problems: (a) how to set the quantitative relationship between the heartbeat timeout judgment threshold T and the heartbeat refresh cycle t of the computing node, so as to promptly identify real node faults while tolerating instantaneous network jitter; (b) how to implement differentiated lifetime management for task records in different lifecycle states, balancing storage efficiency and historical record availability; (c) how to trigger real-time compilation optimization of the inference engine through pre-warm-up inference, reducing the startup latency of the first formal inference; and (d) how to optionally push task progress information to users in real time without affecting the core scheduling logic.

[0013] To address the shortcomings of existing technologies, this invention provides a distributed artificial intelligence inference task scheduling system and method for heterogeneous computing power. The system solves the aforementioned technical problems by means of techniques such as separating scheduling and computation, type-aware distribution of subdivided task queues, competitive acquisition with dual resource gating, dynamic pipeline adaptation with memory awareness, asynchronous heartbeat fault transfer, and differentiated storage expiration strategies.

[0014] The first aspect of this invention provides a distributed artificial intelligence inference task scheduling system for heterogeneous computing power, including a scheduling service module, a resource-aware computing execution module, and a distributed cache module. The three modules work together to complete the distribution, execution, monitoring, and recovery of tasks.

[0015] The second aspect of the present invention provides a distributed artificial intelligence inference task scheduling control method for heterogeneous computing power. The method runs on a scheduling service node without a graphics processor and sequentially executes the steps of request access, type-oriented queuing, node status inspection, and suspended task rescue.

[0016] The third aspect of this invention provides a distributed artificial intelligence inference task computing end control method for heterogeneous computing power. The method runs on a computing node configured with a graphics processor and sequentially executes dual resource gating, resident type priority routing, memory-aware pipeline adaptation, and stateless inference execution steps. It also executes the asynchronous heartbeat maintenance step in parallel through an independent asynchronous daemon thread. The above-mentioned scheduling end control method and computing end control method run in concert with distributed cache as the only state synchronization medium to jointly realize load distribution guided by resident model type and minimize model switching overhead.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described in the second or third aspect above.

[0018] The beneficial technical effects of this invention include:

[0019] (1) Complete separation of scheduling and computation: The scheduling service module does not need to be configured with a graphics processor, reducing hardware deployment costs; the inference load does not affect the scheduling response performance, improving the overall system throughput.

[0020] (2) Type-aware distribution of subdivided task queues: The scheduling service module distributes task identifiers to the corresponding subdivided task queues according to the task type. The computing nodes obtain tasks from the subdivided queues that match their current resident model through the resident type priority routing submodule, so as to minimize unnecessary model switching.

[0021] (3) Dual resource gating mechanism: The computing node can only initiate a request for acquisition when both the concurrent task count and the graphics processor memory reserve meet the conditions. The acquisition frequency is automatically adjusted according to its own real-time computing power. There is no need to manually configure the node weight parameters, which is suitable for heterogeneous computing power environments.

[0022] (4) Memory-aware dynamic pipeline adaptation and preheating optimization: When switching models, preheating inference is performed to trigger the inference engine to compile and optimize in real time. Subsequent formal inference directly uses the optimized computation graph to perform, reducing the initial inference startup latency.

[0023] (5) Double threshold heartbeat failover: The threshold design of T>2t tolerates instantaneous network jitter; during failover, the task is rewritten to the head of the subdivision queue first, and the progressive retry counter is used to prevent abnormal tasks from consuming resources indefinitely.

[0024] (6) Differentiated expiration strategy: Set different lifespans for cached data based on task status to achieve a balance between storage efficiency and historical availability.

[0025] The system, scheduling control method, computing control method, and computer-readable storage medium provided by this invention all share the following specific technical features: a three-way collaborative scheduling mechanism based on independently established subdivided task queues according to task type, a task metadata hash table, and a node heartbeat timestamp registry. This specific technical feature constitutes the core contribution that distinguishes this invention from the prior art, making the system, method, and storage medium of this invention belong to a single overall inventive concept. Attached Figure Description

[0026] Figure 1 This is a system architecture diagram of the present invention, showing the composition of the scheduling service module, the resource-aware computing and execution module, the distributed cache module, and the persistent storage area, as well as their interaction with the user terminal;

[0027] Figure 2 This is a flowchart of the scheduling terminal control method of the present invention, showing the four steps executed by the scheduling service node: request access, classification and targeted queuing, node status inspection, and suspended task rescue.

[0028] Figure 3 This is a flowchart of the computing terminal control method of the present invention, showing the five steps executed by the computing node: dual resource gating, resident type priority routing, memory-aware pipeline adaptation, stateless inference execution, and asynchronous heartbeat maintenance.

[0029] Figure 4 This is a schematic diagram of the heartbeat keep-alive and suspended task rescue mechanism of the present invention, showing the three-way interaction of the heartbeat monitoring mechanism and the suspended task rescue and retry process;

[0030] Figure 5This is a schematic diagram of the task state transition and differentiated expiration strategy of the present invention, showing the transition relationship between various task states and the corresponding lifespan settings;

[0031] Figure 6 This is a schematic diagram of the priority-aware queue routing of the present invention, showing the priority construction logic of the atomic blocking pop instruction and the two routing results of hit and demotion. Detailed Implementation

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0033] The main terms used in this specification are defined as follows:

[0034] Artificial intelligence reasoning (also referred to as AI reasoning in this invention): refers to the process of using a trained artificial intelligence model to perform forward computation on input data to produce output results.

[0035] The artificial intelligence inference tasks scheduled by this invention include, but are not limited to, the following typical types: (a) face effect processing tasks, where the input data is an image file (JPEG / PNG, etc.) containing a face region. The model locates the coordinates of key facial points in the image and applies feature-level or pixel-level transformation operations to the corresponding pixel regions, outputting a processed image file; (b) video rendering tasks, where the input data is a sequence of video frames. The model extracts spatial features from each frame and performs reconstruction or enhancement calculations, outputting a processed video file; (c) image generation tasks, where the input data is text prompts or reference images. The model performs forward inference based on trained generation weights and outputs an image file that satisfies the input description. The input data types for the above tasks are identified in the task parameters through a task type field. The computing node loads the corresponding model according to the task type and performs forward calculations on the input data, and the output results are uploaded to the persistent storage area. This invention does not involve the training or structural modification of the above models. The models used in each task type are all known artificial intelligence models in this technical field.

[0036] Subdivided task queues: These are collections of independent task queues established according to task type. Different types of AI inference tasks are written into their corresponding subdivided queues to support type-aware priority acquisition by computing nodes.

[0037] Resident model: refers to a model that is currently loaded into the graphics processor memory of the compute node and is in an available state. The compute node records its type identifier and model object handle through the local model index.

[0038] Warm-up inference: refers to a complete inference path executed with all-zero tensors or minimum-specification input data after the model is loaded. It is used to trigger the just-in-time compilation optimization of the inference engine and verify the integrity of the model.

[0039] Dual resource gating: This refers to a control mechanism whereby a compute node simultaneously checks two resource indicators—the concurrent task count and the graphics processor's (GPU) memory availability—before initiating a task acquisition request. The GPU memory availability is determined based on the minimum GPU memory threshold required for each task type, which is pre-set independently for each task type through a system configuration file.

[0040] Heartbeat timestamp: refers to the time stamp when a compute node periodically writes to the distributed cache. The scheduling service node determines the liveness status of the compute node by detecting this timestamp.

[0041] Time to Live (TTL) refers to the validity period of data in a distributed cache. Data is automatically deleted after this time. This invention utilizes this mechanism to achieve automatic cleanup of task data and timeout detection of node status.

[0042] Independent temporary workspace: refers to a temporary directory independently allocated for each task identifier in the local file system of the compute node. It is used to store downloaded input resources, intermediate files and inference output during task execution. The entire directory is cleaned up and released after the task is completed or fails, in order to maintain the stateless nature of the compute node.

[0043] Persistent storage area: refers to a remote storage service that is not local to the compute node and can still be accessed after the node restarts or fails. In this embodiment, it is specifically an object storage space used to save the result files of inference output and the archived logs of permanently failed tasks.

[0044] [Example 1] Distributed Artificial Intelligence Inference Task Scheduling System for Heterogeneous Computing Power

[0045] See Figure 1 The system provided by this invention comprises three core components: a scheduling service module, a resource-aware computing execution module, and a distributed caching module. The scheduling service module is deployed on Linux server nodes without graphics processors; the resource-aware computing execution module is deployed on computing nodes equipped with graphics processors, and multiple computing nodes can be deployed to form a heterogeneous cluster; the distributed caching module is implemented using Redis and deployed as an independent service. The system also relies on an external persistent storage area to complete the acquisition of input resources and the storage of inference results and archived logs. In this embodiment, this is specifically an object storage space, which is an external dependent component of the system and not part of the invention itself.

[0046] I. Scheduling Service Module

[0047] The scheduling service module includes the following functional sub-modules:

[0048] (a) Requesting access to the submodule

[0049] Under the premise of obtaining explicit authorization from the data subject of the task and complying with the provisions of the Personal Information Protection Law of the People's Republic of China and other relevant laws and regulations on personal information protection, the system receives AI inference task requests submitted by users via the HTTP protocol; verifies the user's access token by authenticating the request; parses the request parameters and extracts the task type identifier and input file path information; generates a globally unique task identifier for the verified task, which in this embodiment adopts the Universally Unique Identifier (UUID) format and is generated based on a combination of timestamp and random number; serializes the task parameters into the distributed cache module using a hash structure, and the fields include: task type identifier, input file path, resource file path, processing parameters, submission time, submitting user, initial status (waiting), and retry counter (initial value 0); sets the first expiration time according to the initial status of the task (waiting), which is 6 hours in this embodiment.

[0050] (II) Sub-module for type-based targeted team entry

[0051] This submodule is the key technical feature that distinguishes this invention from the existing technology that uses a single global task queue scheme.

[0052] This submodule parses the task type field in the task parameters and writes the task identifier into the corresponding sub-task queue in the distributed cache module using an atomic push instruction (specifically, the Redis LPUSH command in this embodiment). Task identifiers for different task types are written to their respective independent sub-queues; for example, face effects processing tasks are written to the face queue, and video rendering tasks are written to the video queue. Each type of task is stored independently and does not share a common queue.

[0053] The establishment of the subdivided task queue set provides the foundation for the subsequent resource-aware computation execution module to implement residency type-priority routing: compute nodes can list subdivided queue identifiers in priority order in the atomic blocking pop-up instruction, thereby prioritizing the acquisition of tasks that match their current residency model. See also Figure 6 When a node resides in a face effects processing model, its atomic blocking pop-up instruction places the face queue identifier first, ensuring that it can always be hit first when the face queue is not empty, thus avoiding unnecessary model switching.

[0054] This submodule also provides a task queue length query interface, allowing the monitoring system to understand the backlog of various types of tasks in real time.

[0055] (III) Node Status Guardian Submodule

[0056] An independent thread is started to poll the node heartbeat timestamp registry in the distributed cache module at a preset frequency (once every 60 seconds in this embodiment); all computing nodes in the registry are traversed, the heartbeat timestamp of each node is read, and the difference between the current time and the heartbeat timestamp is calculated; when the difference exceeds the heartbeat timeout judgment threshold T, the node is judged to have timed out, and the suspended task rescue submodule is triggered.

[0057] (iv) Suspend the mission rescue submodule

[0058] When the node status monitoring submodule detects a heartbeat timeout node, this submodule executes the following steps in sequence: marking the compute node as offline; traversing the task metadata hash table, filtering for task identifiers whose execution node field matches the offline node and whose status is "in execution"; rolling back the task status to "waiting", clearing the execution node field, and performing an atomic increment operation on the retry counter in the task parameters; if the incremented retry counter does not exceed the preset maximum retry threshold (3 times in this embodiment), the task identifier is written to the head of the corresponding sub-task queue through an atomic push instruction (specifically, the Redis LPUSH command in this embodiment), so that it can be obtained by other normal compute nodes first; if the incremented retry counter exceeds the preset maximum retry threshold, the task identifier is marked as permanently failed, no longer written to the task queue, and the failure log is persistently archived; in some embodiments, an operation and maintenance alarm may also be triggered.

[0059] The incremental retry counter design prevents infinite retries caused by fundamental problems in the task itself (such as corrupted input data or model incompatibility), and avoids the continuous consumption of system resources by abnormal tasks.

[0060] (v) Progress query submodule (optional)

[0061] The progress query submodule is an optional functional component of the scheduling service module. It can be selectively deployed according to business needs without affecting the operation of the system's core scheduling logic. This submodule reads the progress percentage field, stage description field, and estimated remaining time field from the task metadata hash table of the distributed cache module in real time, and pushes the progress information to the user terminal via HTTP interface or WebSocket protocol, allowing users to understand the task execution progress in real time.

[0062] II. Resource-Aware Computing Execution Module

[0063] The resource-aware computing execution module is deployed on compute nodes equipped with graphics processors and includes the following functional sub-modules and a task execution thread pool. There can be multiple compute nodes, which together form a heterogeneous cluster with varying computing power.

[0064] (a) Dual Resource Gating Submodule

[0065] Before initiating a task acquisition request, this submodule simultaneously checks the following two resource metrics:

[0066] Concurrent task count detection: This submodule uses the number of active threads in the task execution thread pool as the concurrent task count indicator; each task is acquired by one thread, and the thread is released when the task is completed or fails; when the number of active threads reaches the maximum number of threads in the task execution thread pool, it is determined that the concurrent task count does not meet the condition.

[0067] Graphics processor memory availability detection: Call the graphics processor status query interface to obtain the current memory usage and total capacity; compare the remaining memory with the minimum memory threshold corresponding to the current task type to be obtained. The minimum memory threshold for each task type is set independently in advance through the system configuration file; if the remaining memory is lower than the threshold, it is determined that the memory availability condition is not met.

[0068] This submodule is only allowed to send a task retrieval request to the distributed cache module when both the concurrent task count and the available GPU memory meet preset conditions. The effect of this dual resource gating is that the task retrieval frequency of a computing node is automatically matched to its real-time computing power—nodes with strong computing power complete tasks faster and release resources more quickly, naturally acquiring more tasks; nodes with weak computing power acquire fewer tasks. This mechanism eliminates the need for manual configuration of the weight parameters for each node, achieving automatic load balancing in heterogeneous computing environments.

[0069] (ii) Preferred routing submodule based on dwell type

[0070] See Figure 6 This submodule constructs and sends an atomic blocking pop command (specifically, the Redis BLPOP command in this embodiment) to the distributed cache module. The key to this submodule lies in the construction of the command parameters: the command parameters contain multiple sub-task queue identifiers, among which the sub-task queue identifier corresponding to the current resident model type of this node is placed first with the highest priority, and the remaining sub-task queue identifiers are arranged in order.

[0071] The distributed caching module checks each sub-task queue sequentially according to the order of the instruction parameters, returning the head task identifier of the first non-empty queue. The atomic blocking pop instruction is atomic: when multiple compute nodes send requests simultaneously, the same task identifier will only be obtained by one compute node, achieving mutual exclusion of task allocation.

[0072] The blocking timeout for the atomic blocking pop-up command is set to 1 second. After the timeout, this submodule will re-trigger the dual resource gating submodule's detection and re-initiate the request, enabling the node to respond to the system stop signal in a timely manner.

[0073] Through the above-mentioned residency type priority routing mechanism, when there is a task in the system that matches the current residency model type of the compute node, the node will prioritize obtaining this type of task without performing model switching, and inference can start immediately; only when the corresponding sub-queue is empty will the node obtain other types of tasks from other sub-queues and perform pipeline switching.

[0074] (III) Memory Sensing Pipeline Adaptation Submodule

[0075] The overall processing logic of this submodule and Figure 3 The adaptation steps of the memory-aware pipeline are consistent with those in the flowchart of the computing-side control method shown. This submodule maintains a local model index, which is an in-memory data structure that records: the identifier of the currently resident model type, the handle of the model object (a memory reference pointing to the loaded model), and the list of local model libraries (recording the downloaded model files and their corresponding task types).

[0076] After obtaining the task identifier, this submodule reads the task parameters from the distributed cache module, parses the task type field, and compares it with the currently resident model type recorded in the local model index:

[0077] In scenario one, when the types are the same, this submodule directly reuses the resident model inference pipeline and notifies the stateless inference execution submodule to start execution immediately, minimizing the inference startup delay.

[0078] Scenario 2: When the types are inconsistent, this submodule executes the following model switching process:

[0079] Step 1: Set the current model object handle to null, explicitly call the runtime environment garbage collection mechanism, and call the graphics processor's memory release interface to ensure that the memory space occupied by the model is completely released.

[0080] Step 2: Check if the target type model weight file exists in the local model library list; if it exists, load it directly from the local machine; if it does not exist, download it from the model distribution server via network request, save it to the local model library, and then load it.

[0081] Step 3: Load the model weights into the graphics processor's memory and initialize the runtime context of the inference engine;

[0082] Step four, perform pre-inference—execute a complete inference path once with all-zero tensors or minimum-size input data. This pre-inference simultaneously triggers the inference engine to perform real-time compilation and optimization of the computation graph; verifies the integrity of the loaded model weight file and the availability of the inference path; and enables subsequent formal inference tasks to directly use the compiled and optimized computation graph, reducing the startup latency of the first formal inference.

[0083] Step 5: Update the target model type identifier to the currently resident model type field of the local model index.

[0084] (iv) Stateless reasoning execution submodule

[0085] Based on the input file path recorded in the task parameters, the resource files required for the task are downloaded from the preset persistent storage area to the local independent temporary workspace (i.e., the temporary directory independently allocated by this task in the local file system). The loaded model inference pipeline is called to perform artificial intelligence inference calculations. The inference process is executed by the graphics processor.

[0086] During the inference process, this submodule writes the intermediate status of the task to the task metadata hash table of the distributed cache module in real time. The intermediate status includes: progress percentage field (value range 0 to 100), current processing stage description field, and estimated remaining time field (dynamically calculated based on the ratio of completed progress percentage to consumed time).

[0087] After inference is complete, this submodule uploads the result file to a pre-defined persistent storage area (specifically, object storage space in this embodiment), where the scheduling service module manages the storage and distribution of the result file. Subsequently, all files in the local independent temporary workspace (including downloaded resource files, intermediate processing results, and log files) are cleaned up, restoring the compute node to its initial usable state, achieving a stateless design. This stateless design allows compute nodes to join or leave the system at any time without state migration, significantly reducing operational complexity.

[0088] The task input data processed by this submodule must be collected and used only with the explicit informed consent of the data subject and in accordance with the provisions of the Personal Information Protection Law of the People's Republic of China and other relevant laws and regulations on personal information protection. This system is not responsible for data processing beyond the legitimate reasoning and calculation purposes described in this invention, nor does it provide data processing interfaces beyond these purposes.

[0089] (v) Asynchronous heartbeat maintenance submodule

[0090] An independent asynchronous daemon thread is started. This thread runs in parallel with the task execution thread without interfering with each other. The daemon thread periodically executes the following at a period t (t=30 seconds in this embodiment): refreshes the time-to-live (TTL) of the task lock in the distributed cache module (set to 90 seconds in this embodiment), and updates the heartbeat timestamp of the node in the node heartbeat timestamp registry, maintaining the node's online status throughout the task execution.

[0091] The maximum number of threads in the task execution thread pool is set through a configuration file; in this example, the default value is 4. Each task is executed in an independent thread, and the threads are isolated from each other. An exception in a single task will not affect the execution of other tasks.

[0092] III. Distributed Cache Module

[0093] The distributed caching module is implemented using Redis and deployed as an independent service, serving as the sole state synchronization medium between the scheduling service module and the resource-aware computing execution module. This module maintains the following four types of data structures:

[0094] Subdivided task queues: Multiple independent list structures are established according to task type, supporting atomic writes from the head (corresponding to the Redis LPUSH command) and atomic blocking pops (corresponding to the Redis BLPOP command); when multiple computing nodes send retrieval requests simultaneously, the distributed cache ensures that the same task identifier is retrieved by only one node, realizing mutual exclusion of task allocation; each subdivided queue schedules tasks in first-in-first-out order.

[0095] Task metadata hash table: Using the task identifier as the key, it stores key-value pairs of fields such as task parameters, task status, execution node identifier, progress information, retry counter, creation time, and update time; each record has a Time-to-Live (TTL) set according to the task status (see Example 5 for details).

[0096] Node heartbeat timestamp registry: Using the node identifier as the key, the registry maintains the heartbeat timestamp of each computing node through atomic writing and setting time-to-live instructions, thereby enabling automatic failure detection of heartbeat timeout.

[0097] Node configuration registry: Stores the name, network address, maximum concurrency, graphics processor model, and supported task type configuration information of each computing node in a hash structure. In some embodiments, the registry can be read by the scheduling service module in the following scenarios: (1) When the scheduling service module needs to verify whether a task type is supported by a corresponding computing node, it reads the task type field supported by each node to filter invalid task types; (2) When the scheduling service module needs to determine the current carrying capacity of the target queue before restoring the task write-back, it reads the maximum concurrency number field of the node; (3) The monitoring system queries the cluster topology information through the node configuration registry and displays the online status and configuration overview of each node in conjunction with the node heartbeat timestamp registry.

[0098] [Example 2] Dispatch Terminal Control Method

[0099] See Figure 2 The scheduling control method runs on the server node without a graphics processor where the scheduling service module is located, and includes the following steps:

[0100] Step S101, Access Request Step: Under the premise of obtaining explicit authorization from the task data subject and complying with the provisions of the Personal Information Protection Law of the People's Republic of China and other relevant laws and regulations on personal information protection, receive the artificial intelligence inference task request submitted by the user through the HTTP protocol; authenticate the request and verify the user's access token; parse the request parameters and extract the task type identifier and input file path information; generate a globally unique task identifier for the verified task; write the task parameters into a distributed cache in a hash structure, wherein the task parameters include a retry counter with an initial value of zero; set the first expiration time according to the initial state of the task (waiting).

[0101] Step S102, type-oriented enqueueing step: parse the task type field in the task parameters, and write the task identifier into the sub-task queue corresponding to the task type through atomic push instructions. The task identifiers of different task types are written into their respective independent sub-task queues.

[0102] Step S103, Node status inspection step: Poll the heartbeat timestamp of each computing node in the distributed cache at a preset frequency (once every 60 seconds in this embodiment), and identify timeout nodes whose heartbeat interval exceeds the threshold T based on the difference between the current time and the heartbeat timestamp; where T satisfies T>2t, and t is the heartbeat refresh cycle of the computing node.

[0103] Step S104, Suspended Task Rescue Steps: For the identified timeout nodes, retrieve the executing task identifier they hold; for each retrieved task, roll back its task status to waiting, and atomically increment the retry counter in the task parameters; if the incremented retry counter does not exceed the preset maximum retry threshold, write the task identifier to the head of the corresponding sub-task queue through an atomic push instruction to wait for execution again; if the incremented retry counter exceeds the preset maximum retry threshold, mark the task as permanently failed, no longer write it to the task queue, and persist the failure log to archive.

[0104] The scheduling control method is continuously executed in a loop on the scheduling service node as an independent thread or independent process. The request access and classification-oriented queuing steps are driven by user requests, and the node status inspection and suspended task rescue steps are periodically triggered by timers.

[0105] [Example 3] Computational Terminal Control Method

[0106] See Figure 3 The computing-side control method runs on a computing node equipped with a graphics processor, where the resource-aware computing execution module resides, and includes the following steps:

[0107] Step S201, Dual Resource Gating Step: Before initiating a task acquisition request, simultaneously check whether the number of active threads in the local task execution thread pool is lower than the preset upper limit and whether the remaining video memory of the graphics processor is not lower than the preset threshold. The subsequent steps are executed only when both conditions are met. The minimum video memory threshold required for each task type is set independently in advance through the system configuration file.

[0108] Step S202, Residency Type Priority Routing Step: Send an atomic blocking pop instruction containing multiple sub-task queue identifiers to the distributed cache; the sub-task queue identifier corresponding to the current residency model type of this node is placed first with the first priority, and the remaining sub-task queue identifiers are arranged in order; the distributed cache checks each sub-task queue in the order of the instruction parameters and returns the head task identifier of the first non-empty queue.

[0109] Step S203, Memory-Aware Pipeline Adaptation Step: Read the task type field in the task parameters and compare it with the currently resident model type; if the required model type is consistent, directly reuse the resident model inference pipeline to proceed to step S204; if inconsistent, execute the following sub-steps in sequence: (a) Set the current model object handle to null and call the graphics processor's memory release interface; (b1) Obtain the target model weight file from the local model library or remote model distribution server as needed (if the corresponding type of weight file already exists in the local model library, load it directly; otherwise, download it from the model distribution server via the network, save it to the local model library, and then load it); (b2) Load the weight file into the graphics processor's memory and initialize the runtime context of the inference engine; (c) Execute a complete inference path once with all-zero tensors or minimum specification input data as a warm-up inference to trigger the inference engine's just-in-time compilation optimization of the computation graph; (d) Update the target model type identifier to the currently resident model type field of the local model index.

[0110] Step S204, Stateless Inference Execution Steps: With explicit authorization from the task data subject and in compliance with the relevant laws and regulations on personal information protection, such as the "Personal Information Protection Law of the People's Republic of China," the task downloads the required resource files from the preset persistent storage area in the temporary workspace independently allocated for this task, based on the input file path recorded in the task parameters, and executes artificial intelligence model inference. During inference, intermediate states including progress percentage, stage description, and estimated remaining time are written to the distributed cache in real time. After inference is completed, the result file is uploaded to the preset persistent storage area, and all files in the local independent temporary workspace are cleaned up, restoring the computing node to its initial usable state.

[0111] The collection and use of input data processed in this step must comply with the aforementioned compliance requirements. This system is not responsible for data processing beyond the legitimate reasoning and calculation purposes described in this invention, nor does it provide data processing interfaces beyond these purposes.

[0112] Step S205, Asynchronous heartbeat maintenance step: The task lock's time-to-live (TTL) in the distributed cache is periodically refreshed and the node's heartbeat timestamp is updated by an independent asynchronous daemon thread at a period of t (t=30 seconds in this embodiment), thus maintaining the node's online status throughout the task execution.

[0113] After the compute node starts, it continuously executes steps S201 to S204 in a loop. The asynchronous heartbeat maintenance step S205 runs in parallel in an independent daemon thread and does not block the main thread.

[0114] When the scheduling control method (Example 2) and the computing control method (Example 3) run in tandem, the distributed cache is used as the sole state synchronization medium: the scheduling service node atomically writes the task identifier into the corresponding sub-task queue through the type-oriented enqueue step; the computing node competitively acquires the task with the sub-task queue corresponding to the current resident model type as the first priority, thereby realizing load allocation guided by the resident model type at the system level, minimizing model switching overhead, and improving the overall throughput efficiency of the heterogeneous computing power cluster.

[0115] [Example 4] Heartbeat Dual Threshold Design

[0116] See Figure 4 In this invention, the threshold T for determining heart rate timeout is set to satisfy the relationship T>2t. In this embodiment, t=30 seconds and T=90 seconds.

[0117] The technical principle of the T>2t design is as follows: Computational nodes periodically write heartbeat timestamps according to a cycle t. This writing operation involves a certain delay due to network transmission. The scheduling service node reads the heartbeat timestamps during a polling cycle to determine the timing. If network congestion occurs momentarily, it may read an old timestamp from the previous cycle, or the latest heartbeat may not yet have been written. Setting the timeout threshold to more than twice the heartbeat cycle ensures that under normal network conditions, the scheduling service node can detect at least one valid heartbeat update, thus tolerating heartbeat delays caused by momentary network fluctuations and avoiding misjudging normal nodes as timeouts or offline.

[0118] [Example 5] Differentiated Expiration Strategy

[0119] See Figure 5 This invention sets the Time-to-Live (TTL) for each record in the task metadata hash table of the distributed cache module according to the differences in task status, thereby achieving efficient utilization of storage resources.

[0120] Waiting status: Set the first expiration time (6 hours in this example). If a task is not acquired by the computing node after 6 hours, it indicates that there may be an anomaly in the system. In this case, it will automatically expire and be cleaned up to avoid the backlog of invalid tasks.

[0121] In execution status: A second expiration time is set (2 hours in this embodiment), which is shorter than the first expiration time. A normal single inference task should not exceed 2 hours. If it remains in the execution state after this time, it is highly likely that the computing node has experienced an extreme failure such as a power outage, causing the heartbeat mechanism to fail to detect it in time. In this case, the task data will be automatically cleaned up after expiration. The second expiration time is strictly shorter than the first expiration time, which helps to accelerate the identification of abnormally suspended tasks.

[0122] Completed Status: A third expiration period is set (7 days in this example), which is longer than the first expiration period. Completed tasks are the data that users care about most, and the 7-day retention period provides users with ample query opportunities.

[0123] Failure status: A fourth expiration period is set (3 days in this embodiment), which is longer than the first expiration period and shorter than the third expiration period. Failed tasks are retained for maintenance personnel to troubleshoot and analyze problems.

[0124] Permanent failure status: No expiration time is set, and persistent archive storage is performed as a basis for system operation history and long-term problem analysis.

[0125] [Example 6] Computer-readable storage medium

[0126] This embodiment provides a computer-readable storage medium, which includes, but is not limited to, read-only memory, random access memory, disk, optical disk, solid-state drive, etc.; the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the scheduling terminal control method described in Embodiment 2 or the steps of the computing terminal control method described in Embodiment 3.

[0127] In the theoretical analysis, the system provided by this invention has the following technical effects compared with the existing solution that uses a single global task queue: (i) In the scenario of concentrated submission of tasks of the same type (i.e., the vast majority of tasks in the queue match the model type of the computing node), due to the priority routing of the residency type, the node hardly needs to perform model switching. The improvement of the overall inference throughput of the cluster is positively correlated with the ratio of the model switching time to the total time of a single task. Taking face effect processing tasks as an example, if the model loading time accounts for about 50% of the total time of a single task, then under the above scenario assumption, the overall throughput of the cluster can theoretically be improved by more than 50% (the above estimate is a theoretical calculation based on task structure analysis. The actual improvement varies depending on the task type, hardware configuration and load distribution, and has not been verified by actual testing under a specific hardware environment). (ii) In the scenario of node failure, the longest waiting time for the suspended task to recover does not exceed T plus the node status inspection and polling interval (about 90 seconds + 60 seconds = 150 seconds in this embodiment). Compared with the existing solution that relies on manual intervention, the average task recovery time can be reduced from minutes to seconds.

[0128] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can be modified and varied in various ways. For example, the distributed cache can be implemented using methods other than Redis; the operating system of the computing nodes can be Linux or other operating systems; the number and type of subdivided task queues can be flexibly configured according to actual business needs; the memory threshold for each task type can be independently adjusted through a configuration file; the heartbeat period and timeout threshold can be adjusted according to the actual network environment. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed artificial intelligence inference task scheduling system for heterogeneous computing power, comprising a scheduling service module, a resource-aware computing execution module, and a distributed caching module, characterized in that: The scheduling service module runs on a server node without a graphics processor and includes a request access submodule, a type-oriented queuing submodule, a node status monitoring submodule, and a suspended task rescue submodule. It is used to receive and verify task requests, generate a unique task identifier, write the task parameters containing a zero-value retry counter into a distributed cache module in a hash structure and set the lifespan according to the initial state, atomically write the task identifier into the subdivided task queue according to type, detect heartbeat timestamps to identify timeout nodes, roll back the executing tasks of the node and increment the retry counter, write it back to the head of the queue if it does not exceed the retry threshold, and mark it as permanently failed and archive it if it exceeds the threshold. The resource-aware computing execution module runs on a node cluster configured with a graphics processor. Each node includes a task execution thread pool and a dual resource gating submodule, a resident type priority routing submodule, a memory-aware pipeline adaptation submodule, a stateless inference execution submodule, and an asynchronous heartbeat maintenance submodule. It is used to acquire tasks when the number of active threads and the remaining memory both meet the threshold. The memory threshold for each type is set independently by the configuration file. It prioritizes the atomic blocking of the queue corresponding to the current resident model type to pop the task identifier, reuses or switches the inference pipeline according to type and warms it up, executes inference in an independent work area, writes intermediate states containing progress percentage, stage description and estimated remaining time in real time, uploads results and cleans up files, and refreshes the task lock TTL and heartbeat timestamp. The distributed cache module serves as the sole state synchronization medium, maintaining a set of subdivided task queues established by type, a task metadata hash table with differentiated time-to-live (TTL) settings based on task state, and a node heartbeat timestamp registry maintained by atomically writing and setting TTL.

2. The system according to claim 1, characterized in that, The node status protection submodule sets the heartbeat timeout determination threshold to T, and the heartbeat refresh cycle of the asynchronous heartbeat maintenance submodule is t, where T>2t.

3. The system according to claim 1, characterized in that, In the distributed caching module, the lifespan of each record in the task metadata hash table is set differently based on the task status: the task record in the waiting state is set with a first expiration time; the task record in the executing state is set with a second expiration time, which is less than the first expiration time. A third expiration period is set for a task record that is in a completed state, and the third expiration period is longer than the first expiration period; Task records in a failed state are given a fourth expiration period, which is greater than the first expiration period and less than the third expiration period; task records in a permanently failed state are not given an expiration period and are stored in a persistent archive.

4. The system according to claim 1, characterized in that, The specific process of the memory-aware pipeline adaptation submodule for performing pre-warm-up inference is as follows: a complete inference path is executed once with all-zero tensors or minimum specification input data, triggering the inference engine to perform real-time compilation optimization of the computation graph; the pre-warm-up inference also verifies the integrity of the loaded model weight file and the availability of the inference path; after the pre-warm-up inference is completed, subsequent formal inference tasks are directly executed using the compiled optimized computation graph, reducing the startup delay of the first formal inference.

5. The system according to claim 1, characterized in that, The maximum number of threads in the task execution thread pool of the resource-aware computing execution module is set through a configuration file. Each task is executed in an independent thread, and the threads are isolated from each other. When the number of active threads in the thread pool reaches the maximum number of threads, the dual resource gating submodule determines that the concurrent task count has reached the preset upper limit and stops sending retrieval requests to the subdivided task queue until a thread completes its release. The scheduling service module also includes a progress query submodule, which is used to read the progress percentage field, stage description field, and estimated remaining time field from the task metadata hash table of the distributed cache module in real time, and push the progress information to the user terminal through an HTTP interface or WebSocket protocol.

6. A distributed artificial intelligence inference task scheduling control method for heterogeneous computing power, the method running on a scheduling service node without a graphics processor, characterized in that, include: Access request steps: Receive the task request submitted by the user, perform identity verification, generate a globally unique task identifier, write the task parameters into the distributed cache in a hash structure, the task parameters include a retry counter with an initial value of zero, and set the corresponding lifespan according to the initial state of the task; The sub-type targeted enqueueing steps are as follows: parse the task type field in the task parameters, and write the task identifier into the sub-task queue that corresponds to the task type through atomic push instructions. The task identifiers of different task types are written into their respective independent sub-task queues. Node status inspection steps: Poll the heartbeat timestamps of each computing node in the distributed cache at a preset frequency, and identify timeout nodes whose heartbeat interval exceeds the threshold T based on the difference between the current time and the heartbeat timestamp, where T>2t, and t is the heartbeat refresh cycle of the computing node; Suspended task rescue steps: For the identified timeout node, retrieve the in-execution task identifier it holds, roll back the task status to waiting, and atomically increment the retry counter in the task parameters; If the accumulated retry counter does not exceed the preset maximum retry threshold, the task identifier is written to the head of the corresponding sub-task queue through an atomic push instruction to wait for execution again. If the accumulated retry counter exceeds the preset maximum retry threshold, the task will be marked as a permanent failure, will no longer be written to the task queue, and the failure log will be persistently archived.

7. The method according to claim 6, characterized in that, The scheduling control method sets a lifespan based on the task status differences, and updates the corresponding record's lifespan synchronously when the status changes at each stage: when the initial task status is "waiting", a first expiration time is set; when it changes to "execution", a second expiration time is updated, which is less than the first expiration time; when it changes to "completed", a third expiration time is updated, which is longer than the first expiration time; when it changes to "failed", a fourth expiration time is updated, which is longer than the first expiration time and less than the third expiration time; when it changes to "permanent failure", no expiration time is set and persistent archiving is performed.

8. A method for controlling the computing end of a distributed artificial intelligence inference task with heterogeneous computing power, the method running on a computing node configured with a graphics processor, characterized in that, include: Dual resource gating steps: Before initiating a task acquisition request, simultaneously check whether the number of active threads in the local task execution thread pool is lower than the preset upper limit and whether the remaining video memory of the graphics processor is not lower than the preset threshold. Only when both conditions are met will the subsequent steps be executed. The minimum video memory threshold required for each task type is set independently in advance through the system configuration file. The residency type priority routing step is as follows: an atomic blocking pop instruction containing multiple sub-task queue identifiers is sent to the distributed cache. The sub-task queue identifier corresponding to the current residency model type of this node is given the first priority and placed first, and the other sub-task queue identifiers are arranged in order. The distributed cache checks each sub-task queue in the order of the instruction parameters and returns the head task identifier of the first non-empty queue. Memory-aware pipeline adaptation steps: Read the task type field in the task parameters and compare it with the currently resident model type. If the required model type is consistent, the resident model inference pipeline is directly reused; if they are inconsistent, the following steps are executed in sequence: release the current memory resources, download and load the target model weight file from the local model library or remote model distribution server to the memory as needed, perform warm-up inference, and update the local model index. Stateless reasoning execution steps: Perform AI model reasoning in an independent temporary workspace. During the reasoning process, intermediate states including progress percentage, stage description and estimated remaining time are written to the distributed cache in real time. After inference is complete, the result file is uploaded to the preset persistent storage area and local temporary files are cleaned up, so that the compute node is restored to its initial usable state. Asynchronous heartbeat maintenance steps: The time-to-live (TTL) of the task lock in the distributed cache is refreshed periodically by an independent asynchronous daemon thread at a period t, and the node heartbeat timestamp is updated to maintain the node's online status throughout the entire task execution.

9. The method according to claim 8, characterized in that, The specific process of performing pre-warm-up inference in the memory-aware pipeline adaptation step is as follows: execute a complete inference path once with all-zero tensors or minimum specification input data to trigger the inference engine to compile and optimize the computation graph in real time; the pre-warm-up inference also verifies the integrity of the loaded model weight file and the availability of the inference path; after the pre-warm-up inference is completed, subsequent formal inference tasks directly use the compiled optimized computation graph to reduce the startup delay of the first formal inference.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 6 to 9.

Images