A model inference task scheduling method and system based on chip computing power inference

Abstract

The application discloses a model inference task scheduling method and system based on chip computing power inference, and belongs to the technical field of chip load balancing. The method is: constructing a real-time computing power state report of a chip computing power cluster; generating corresponding inference request features for an inference task according to preset business time delay requirements and data volume of the inference task; assigning corresponding target chip nodes to the inference task according to the real-time computing power state report, generating an optimal cooperative execution strategy for the target chip nodes to execute the inference task based on a preset acceleration strategy library and the inference request features; and sending the inference task to the corresponding target chip nodes and performing inference acceleration optimization based on the optimal cooperative execution strategy. Therefore, by implementing the application, the problems of poor adaptability between inference tasks and chips, uneven load between chips caused by uneven distribution of inference tasks, and poor execution efficiency of inference tasks in the prior art can be solved.

Description

Technical Field

[0001] This application belongs to the field of chip load balancing technology, specifically relating to a model inference task scheduling method and system based on chip computing power inference. Background Technology

[0002] Currently, when executing inference tasks for artificial intelligence models on chip computing clusters, traditional load balancing strategies such as static round-robin or minimum connection count are typically used to distribute inference tasks to chips. These are combined with basic inference acceleration strategies (such as manual operator fusion and fixed quantization precision) to execute inference tasks on the chips. These algorithms can generally meet the needs of inference tasks in a single task type and homogeneous chip environment. For example, for a pure image classification request stream, round-robin distribution can ensure that the number of tasks on each chip is similar, maintaining load balance between chips.

[0003] However, in real-world production environments, inference tasks vary widely in type and volume, and the hardware characteristics (computing power, memory bandwidth, operator support list) of each chip differ. Furthermore, the business operations corresponding to inference tasks have strict latency constraints. The aforementioned load balancing strategies lack a joint consideration of the inference task type and chip compatibility. When allocating inference tasks to chips, they rely solely on static indicators such as the number of chip cores or fixed weights, ignoring dynamic loads such as the number of currently queued tasks, the remaining time of tasks being executed, and available video memory. This results in some chips being under high load for extended periods while others remain idle, leading to overall low efficiency in chip inference task execution and the inability of the chip computing cluster to achieve load balancing. Summary of the Invention

[0004] This application proposes a model inference task scheduling method and system based on chip computing power inference, which can solve the problems of poor compatibility between inference tasks and chips in the prior art, uneven distribution of inference tasks leading to uneven load among chips, and poor execution efficiency of inference tasks.

[0005] The first aspect of this application provides a method for scheduling model inference tasks based on chip computing power, the method comprising: Based on the real-time operating data, hardware parameters, and average expected latency parameters of each chip node in the chip computing power cluster, a real-time computing power status report of the chip computing power cluster is constructed. Based on the preset business latency requirements and the data volume of the inference task, corresponding inference request features are generated for the inference task; wherein, the inference request features include task type, computing power requirement level, expected completion time limit, recommended chip model and request ID; Based on the real-time computing power status report, a corresponding target chip node is allocated to the inference task. Based on the preset acceleration strategy library and the inference request characteristics, the optimal collaborative execution strategy for the target chip node to execute the inference task is generated. Based on the optimal collaborative execution strategy, the inference task is sent to the corresponding target chip node and inference acceleration optimization is performed.

[0006] The above scheme collects real-time operating data from each chip node to obtain dynamic load information, and acquires its hardware parameters and average expected latency parameters to determine the actual load capacity of the chip node. The constructed real-time computing power status report provides a reliable basis for subsequent load balancing decisions. Structured analysis of inference tasks reveals inference request characteristics related to the computing power required by the inference task, enabling the system to allocate computing resources to the inference task on demand, identify target chip nodes with sufficient computing power and the ability to execute inference tasks quickly, and improve the execution efficiency of inference tasks. Furthermore, by combining a preset acceleration strategy library with the inference request characteristics, the system finds the most suitable acceleration strategy for the target chip node, avoiding situations where the selected chip, although compatible, is severely overloaded, leading to uneven load distribution among chips. Moreover, the strategy matching also considers task type and expected completion time, taking into account the compatibility between the chip and the inference task and the timeliness of task completion, achieving optimal matching of chip computing resources and inference tasks, improving the overall execution efficiency and acceleration effect of inference tasks, while avoiding resource overload or idleness.

[0007] In one possible implementation of the first aspect, a real-time computing power status report of the chip computing power cluster is constructed based on the real-time operating data, hardware parameters, and average expected latency parameters of each chip node in the chip computing power cluster, specifically as follows: The real-time running data is obtained by collecting the number of tasks to be executed, the estimated total time remaining of tasks currently being executed, and the current available video memory for each chip node; The floating-point computing power, integer computing power, memory bandwidth, and list of supported operators of each chip node are collected to obtain the hardware parameters. The historical inference task completion data of each chip node is executed through a preset sliding window, and the average time taken for different types of inference tasks to be executed by different chip models is calculated to obtain the average expected latency parameter for each chip node to execute different types of inference tasks. The real-time operating data, the hardware parameters, and the average expected latency parameters are associated with the corresponding chip nodes and stored in a preset structured database to obtain the real-time computing power status report.

[0008] In the above scheme, the real-time running data can comprehensively reflect the computing power and available resources of the chip node, providing more accurate input for matching the target chip node for inference tasks; the average expected latency parameter reflects the task processing time, providing data support for selecting acceleration strategies that can process inference tasks in a timely manner; and various types of data are associated with chip nodes and stored in a structured database to form a queryable and updatable real-time computing power status report, which facilitates rapid retrieval and dynamic updates by the scheduling system.

[0009] In one possible implementation of the first aspect, based on preset business latency requirements and the data volume of the inference task, corresponding inference request features are generated for the inference task, specifically as follows: The received inference task is parsed to determine the task type, and the data volume of the inference task is calculated based on the calculation method corresponding to the task type. Based on the data volume and the business latency requirements of the inference task, the computing power requirement level of the inference task is determined, and the maximum execution time of the computing power requirement level is taken as the expected completion time of the inference task; wherein, the business latency requirements are determined by the business scenario corresponding to the inference task. Based on the task type, the recommended chip model for the inference task is determined.

[0010] The above solution clarifies the type of inference tasks by analyzing their types, providing a basis for subsequent computing power requirement assessment and chip selection. It also quantifies the expected completion time of the inference task based on the data volume and the business latency requirements, thus determining the urgency of the task. Based on the task type, it determines the recommended chip model, helping to narrow down the range of candidate chips, accelerating the selection process of target chip nodes, and enabling the rapid finding of suitable chip nodes for the inference task.

[0011] In one possible implementation of the first aspect, based on the real-time computing power status report, a corresponding target chip node is allocated to the inference task. Based on a preset acceleration strategy library and the inference request characteristics, an optimal collaborative execution strategy is generated for the target chip node to execute the inference task. Specifically: Remove chip nodes that are in a faulty or isolated state from the real-time computing power status report, and remove chip nodes whose current available video memory is less than a first threshold. Generate a candidate node set based on the remaining chip nodes after removal. Based on the real-time operational data, a corresponding target chip node is allocated to the inference task from the candidate node set; Combining the hardware parameters of the target chip node and the characteristics of the inference request, the optimal acceleration scheme is matched for the inference task from the acceleration strategy library; The optimal collaborative execution strategy is generated based on the optimal acceleration scheme, the request ID, the target chip node ID, and the estimated completion time of the inference task.

[0012] The above scheme first eliminates chip nodes that are unavailable or unable to perform inference tasks, narrowing down the selection range of target chip nodes. Then, based on the chip load and inference request characteristics, it matches suitable target chip nodes for inference tasks while achieving load balancing between chips, avoiding scheduling tasks to overloaded nodes, so that the resulting optimal acceleration scheme is both compatible with hardware capabilities and meets the latency requirements of the task.

[0013] In one possible implementation of the first aspect, based on the real-time runtime data, a corresponding target chip node is allocated from the candidate node set for the inference task, specifically as follows: Based on the real-time running data, the weighted load value of the currently executing tasks is calculated using the current number of tasks to be executed by the candidate node, the estimated total remaining time of the currently executing tasks, and the preset dynamic weights. The candidate node corresponding to the smallest weighted load value is selected as the target chip node for the inference task; If there are multiple minimum weighted load values, then the corresponding target chip node is selected for the inference task according to the task type.

[0014] The above scheme calculates a weighted load value to reflect the current load level of each chip node, avoiding misjudgments of load based solely on a single indicator such as the number of tasks to be processed. Selecting the candidate node with the smallest weighted load value as the target chip node helps achieve better load balancing in the cluster and shortens the queuing time for inference tasks. When multiple smallest weighted load values ​​exist, further selection of the target chip node based on the task type can prioritize the allocation of inference tasks to chips more suitable for that task type, improving the execution efficiency of inference tasks.

[0015] In one possible implementation of the first aspect, combining the hardware parameters of the target chip node and the inference request characteristics, an optimal acceleration scheme is matched for the inference task from the acceleration strategy library, specifically as follows: The combination of the target chip node and the task type is used as the matching target, and all acceleration candidate schemes that meet the matching target are extracted from the acceleration strategy library. Based on the hardware parameters and the expected completion time, the acceleration candidate solutions are screened, and then the acceleration candidate solution with the highest historical success rate is selected as the optimal acceleration solution from the screening results.

[0016] The above approach uses the combination of target chip node and task type as the matching target to ensure that the selected acceleration candidate solutions are applicable to the hardware and task scenario. By combining hardware parameters and expected completion time limits, solutions that do not meet hardware capabilities or time requirements are eliminated, ensuring the feasibility and timeliness of the acceleration solutions.

[0017] One possible implementation of the first aspect also includes: Calculate the GPU memory resource requirements for the inference task based on the data volume and the task type. If the required video memory resources are greater than the currently available video memory of the target chip node, then a corresponding target chip node is reallocated for the inference task from the candidate nodes.

[0018] The above solution performs memory matching before inference tasks are executed, effectively avoiding task failures or severe performance degradation due to insufficient memory. Furthermore, it ensures that inference tasks are always allocated to target chip nodes with sufficient memory resources through reallocation, improving the success rate of inference task execution.

[0019] In one possible implementation of the first aspect, based on the optimal cooperative execution strategy, the inference task is sent to the corresponding target chip node and inference acceleration optimization is performed, specifically as follows: The optimal collaborative execution strategy is converted into corresponding execution instructions, and the execution instructions and the inference task are sent to the target chip node; A batch processing buffer queue is generated locally on the target chip node. When the batch processing buffer queue meets any set condition, the inference tasks in the batch processing buffer queue are packaged into the execution queue of the target chip node for inference acceleration optimization.

[0020] The above solution generates a batch processing buffer queue locally on the target chip node, which can aggregate and package multiple inference tasks for processing, reducing frequent scheduling and improving the utilization efficiency of chip computing resources. When the batch processing buffer queue meets the set conditions, it is packaged into the execution queue for inference acceleration optimization, balancing the throughput advantages of batch processing and the timeliness of task response, and avoiding the impact of excessive waiting on the overall latency.

[0021] In one possible implementation of the first aspect, the setting conditions include the number of inference tasks in the batch processing buffer queue reaching the current batch size, and the waiting time of the batch processing buffer queue exceeding a preset batch processing timeout threshold. The current batch size is determined by the current load status of the target chip node.

[0022] The above scheme provides two conditions that ensure a large number of inference tasks can be processed at once, while preventing inference tasks from waiting too long, thus achieving an effective balance between throughput and latency.

[0023] The second aspect of this application provides a model inference task scheduling system based on chip computing power inference, the system comprising: a chip computing power recording module, an inference request feature generation module, an execution strategy generation module, and an inference task execution module; Among them, the chip computing power recording module is used to construct a real-time computing power status report of the chip computing power cluster based on the real-time operating data, hardware parameters and average expected latency parameters of each chip node in the chip computing power cluster. The inference request feature generation module is used to generate corresponding inference request features for inference tasks based on preset business latency requirements and the data volume of inference tasks; wherein, the inference request features include task type, computing power requirement level, expected completion time limit, recommended chip model and request ID; The execution strategy generation module is used to allocate corresponding target chip nodes to inference tasks based on the real-time computing power status report, and generate the optimal collaborative execution strategy for the target chip nodes to execute inference tasks based on the preset acceleration strategy library and the inference request characteristics. The inference task execution module is used to send the inference task to the corresponding target chip node and perform inference acceleration optimization based on the optimal collaborative execution strategy. Attached Figure Description

[0024] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram illustrating the specific process of a model inference task scheduling method based on chip computing power inference according to an embodiment of this application; Figure 2 This is a detailed structural diagram of a model inference task scheduling system based on chip computing power inference provided in one embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0028] First Embodiment In the fields of power marketing, dispatching, and operation and maintenance, it is often necessary to solve the problems of real-time performance, accuracy, and computing power utilization under mixed loads of various types of AI inference tasks. For example, inference tasks such as power grid operation status monitoring, fault tracing, dispatch plan optimization, transmission channel hidden danger identification, equipment infrared diagnosis, voice intent recognition, and electricity bill question-and-answer robots have different compatibility with various chip types and different requirements for chip computing power resources. Therefore, how to solve the above problems, achieve fast and timely processing of inference tasks, and maintain load balance among chips is the main research objective of the embodiments of this application.

[0029] like Figure 1 As shown, to address the problems of poor compatibility between inference tasks and chips in existing technologies, and uneven distribution of inference tasks leading to uneven load among chips and poor execution efficiency of inference tasks, the first embodiment of this application provides a detailed flowchart of a model inference task scheduling method based on chip computing power inference. This embodiment's model inference task scheduling method based on chip computing power inference includes steps S1 to S4, and is mainly implemented based on a domestic chip computing power cluster. The specific steps are detailed below: Step S1: Based on the real-time operating data, hardware parameters, and average expected latency parameters of each chip node in the chip computing power cluster, construct a real-time computing power status report of the chip computing power cluster.

[0030] This application embodiment first obtains the real-time operating data of each chip node in the domestic chip computing power cluster, including the number of tasks to be executed at present, the estimated total remaining time of the tasks being executed, and the current available video memory of each chip node.

[0031] The hardware parameters are obtained by collecting the floating-point computing power, integer computing power, memory bandwidth, and supported operator list of each chip node.

[0032] Then, the historical inference task completion data of each chip node is executed through a preset sliding window, and the average time taken for different chip models to execute different types of inference tasks is calculated as the average expected delay parameter for each chip node to execute different types of inference tasks.

[0033] For example, for a speech recognition task (a type of inference task) and the chip node executing the task, 50 successful completion records related to speech recognition in the last 10 seconds are obtained. The actual time consumption data of this inference task are [5,6,5,7,5,…]. The average actual time consumption is calculated to be 5.8ms through the sliding window. Then, the average value is updated as the average expected latency parameter for the chip node to execute the speech recognition task.

[0034] Finally, the real-time running data, the hardware parameters, and the average expected latency parameters are associated with the corresponding chip nodes (associated through the chip node IDs) and stored in a preset structured database to obtain a real-time computing power status report of the chip computing power cluster.

[0035] Each data entry in the real-time computing power status report is indexed by the chip node ID, and the overload indicator field reflects the current load status of each chip node (the presence of the overload indicator field indicates a higher chip node load, while the absence of the overload indicator field indicates a lower chip node load).

[0036] Based on the real-time computing power status report, computing power data is provided to support subsequent decisions on maintaining chip load balance.

[0037] Step S2: Based on the preset business latency requirements and the data volume of the inference task, generate corresponding inference request features for the inference task.

[0038] The inference tasks received by domestic chip computing power clusters are mostly of three types: speech recognition, image classification, and large model generation. Therefore, the embodiments of this application focus on these three types of tasks.

[0039] First, the received inference task is parsed to determine which of the three task types mentioned above it belongs to. Then, based on the computation method corresponding to the task type, the data volume of the inference task is calculated. This data volume actually reflects the computational resources required to complete the inference task and is directly linked to the memory resources of the chip node.

[0040] Specifically, the `type` field in the inference task is parsed or matched using rules to determine the corresponding task type. For example, if the keyword "audio" is present, it's classified as speech recognition; if the keyword "image" is present, it's classified as image classification; and if the keyword "text" or "chat" is present, it's classified as large model generation. Based on the task type, the data volume of the inference task is determined. The data volume is calculated in three ways: (1) Calculation method for speech recognition: Based on the inference task, the audio duration and sampling rate are statistically analyzed, and the data volume is determined based on the product of the audio duration and sampling rate. The unit is uniformly recorded according to the original value. (2) Calculation method for image classification: According to the reasoning task, the image width, height and number of channels are counted. The number of channels for RGB images is fixed at 3. The data volume of a single image is determined based on the product of the image width, height and number of channels. The data volume of multiple images is the sum of the data volumes of each single image. (3) Calculation method for generating large models: Based on the inference task, the number of input text tokens and the number of expected output tokens are counted, and the total number of input text tokens and the number of expected output tokens are accumulated to determine the data volume.

[0041] Then, based on the data volume and the business latency requirements of the inference task, the computing power requirement level of the inference task is determined, and the maximum execution time of the computing power requirement level is taken as the expected completion time of the inference task.

[0042] Specifically, the latency requirement is first determined based on the business scenario corresponding to the inference task. If the data volume is small and the latency requirement is short, the corresponding computing power requirement level is higher to complete the inference task as quickly as possible. If the latency requirement is short but the data volume is large, it cannot be directly assumed to be a high computing power requirement level. Therefore, although the time limit is urgent, the large data volume makes it impossible to complete quickly, and forcing it to be at a high level would overwhelm the system. Therefore, the inference task is generally placed at a medium computing power requirement level, and the appropriate chip node is selected by subsequent load balancing weighted selection.

[0043] For example, this application provides three computing power requirement levels, as detailed below: High computing power requirement level: The maximum execution time is <100ms, and the data volume is relatively small; Medium computing power requirement level: The maximum execution time is between 100ms and 300ms, and the data volume is medium; Low computing power requirement level: The maximum execution time is >300ms and the data volume is large.

[0044] Furthermore, embodiments of this application will also consider the task type to find recommended chip models for the inference task. For example, low-power chip nodes are preferred for speech recognition tasks; general-purpose chip nodes are preferred for image classification tasks; and high-performance large-model chip nodes are preferred for large-model generation tasks.

[0045] Then, a unique request ID is generated for the inference task. The task type, computing power requirement level, expected completion time limit, recommended chip model and request ID are combined to generate corresponding inference request features for the inference task, so as to provide task-dimensional data support for subsequent task scheduling and matching.

[0046] Step S3: Based on the real-time computing power status report, allocate corresponding target chip nodes for the inference task, and generate the optimal collaborative execution strategy for the target chip node to execute the inference task based on the preset acceleration strategy library and the inference request characteristics.

[0047] In this step, the inference request characteristics of the inference task are read, and scheduling decisions are made in conjunction with the status of each chip node provided by the real-time computing power status report, so as to match the inference task with a suitable target chip node and the optimal collaborative execution strategy.

[0048] The received inference tasks are integrated into a task queue, and the following implementation steps S3.1 to S3.5 related to chip matching and acceleration strategy extraction are performed in sequence according to the priority corresponding to the computing power requirement level.

[0049] Step S3.1: Remove chip nodes that are in a fault state or isolated state from the real-time computing power status report, and remove chip nodes whose current available video memory is less than a first threshold. Generate a candidate node set based on the remaining chip nodes after removal. The fault state or isolated state reflects that the chip node is unavailable and cannot perform inference tasks, so it is removed.

[0050] Optionally, a first threshold is set as the minimum video memory required for the inference task (related to the data volume and the task type), for example, a large model generation task requires at least 8GB, and an image classification task requires 2GB.

[0051] Step S3.2: First, based on the real-time running data, calculate the weighted load value of the currently executing task by using the number of tasks to be executed at the candidate node, the estimated remaining total time of the currently executing task, and the preset dynamic weight.

[0052] The weighted load value L The calculation formula is: ; in, Q This represents the number of tasks currently pending execution for the candidate node. T The estimated total remaining time for the task currently in progress. W These are preset dynamic weights.

[0053] The dynamic weight W The calculation method is as follows: ; in, The average expected latency parameter for performing the current type of inference task on the candidate node. This is the overload degradation coefficient. If the candidate node has an overload identifier field, the value is 0.2; otherwise, the value is 1.0. This is the slow node correction coefficient. If the candidate node is marked as a slow node (the ratio of the actual time of a chip node to the average expected latency parameter under this task type exceeds the preset ratio threshold), the value is 0.5; otherwise, the value is 1.

[0054] The smaller the weighted load value, the more suitable the corresponding candidate node is for executing the inference request. Therefore, the candidate node with the smallest weighted load value is selected from the candidate node set as the target chip node for the inference task.

[0055] If multiple candidate nodes have the same and lowest weighted load values, the candidate node that best matches the task type is selected as the target chip node for the inference task. For example, a low-power chip node is preferred for a speech recognition task.

[0056] Step S3.3: Using the combination of the target chip node and the task type as the matching target, extract all acceleration candidate solutions that match the matching target from the acceleration strategy library. The acceleration strategy library stores the historical success rates of different acceleration candidate solutions under various combinations of chip nodes and task types.

[0057] Specifically, based on the matching target, all available acceleration candidate schemes are retrieved from the acceleration strategy library, and based on the target chip node's supported operator list, acceleration candidate schemes that are unavailable due to missing operators are eliminated. For example, if the target chip node's supported operator list lacks the BatchMatMul operator, but the INT8 inference task after model quantization requires this operator, then the acceleration candidate scheme for quantized INT8 is marked as unavailable.

[0058] Based on the expected completion time of the inference task, the current load status of the target chip node, and the hardware parameters, the eliminated acceleration candidate schemes are pre-filtered, and the acceleration candidate scheme with the lowest latency is selected first. If the target chip node is currently overloaded or is a slow node, a lightweight acceleration method is selected; if the data volume is extremely small, dynamic batch processing is skipped.

[0059] For the pre-filtered acceleration candidate solutions, the acceleration candidate solution with the highest historical success rate is directly selected as the optimal acceleration solution.

[0060] Furthermore, if the target chip node's list of supported operators is missing the operators required for the inference task, operator fusion is automatically disabled, and the acceleration candidate with the lowest software simulation overhead is selected; if the expected completion time is less than 1.5 times the average expected latency parameter, the acceleration candidate with the lowest latency is selected first.

[0061] Step S3.4: Calculate the estimated completion time of the reasoning task. The calculation formula is as follows: ; in, For the current time, The average expected latency parameter for performing the current inference task on the target chip node. This represents the number of tasks currently pending execution on the target chip node.

[0062] If the estimated completion time is compared with the expected completion time limit, and the result significantly exceeds the preset time limit threshold, an early warning flag is triggered for the inference request, and the inference request is given special attention.

[0063] Step S3.5: Based on the optimal acceleration scheme, the request ID, the target chip node ID, the estimated completion time of the inference task, and the warning flag (if triggered), generate the optimal collaborative execution strategy for the target chip node to execute the inference task.

[0064] As an improvement to the above solution, this embodiment of the application, before distributing the inference task to the target chip node, will also reserve computing resources for the inference task by detecting the memory status of the target chip node, specifically including the following steps: (1) Integrate the optimal collaborative execution strategies corresponding to each inference task into a collaborative execution strategy table. Group all inference tasks in the collaborative execution strategy table according to the ID of the target chip node. Inference tasks belonging to the same target chip node are assigned to the same group. Then, the inference tasks in the same group are grouped into corresponding queues to be distributed. (2) For each inference task in the distribution queue, perform the following reserved computing resource operations: Based on the data volume and task type of the inference task, calculate the GPU memory resource requirements of the inference task, including model weights, intermediate activation data, key-value cache, etc.

[0065] In the currently available video memory of the target chip node, the video memory value corresponding to the video memory resource requirement of the inference task is pre-locked, and the currently available video memory field in the real-time computing power status report is updated, that is, the video memory value reserved this time is subtracted from the currently available video memory. If it is found that the currently available video memory of the target chip node is insufficient during the above reservation process (for example, due to instantaneous resource contention caused by concurrent reservation, the remaining currently available video memory field after subtraction is negative), the inference request is marked as "resource reservation failed", and the corresponding target chip node is reallocated to the inference task from the candidate nodes.

[0066] In existing technologies, traditional load balancing and acceleration strategies are independent of each other, resulting in selected chips with low load but lacking support for the required operators, or selected chips that are compatible but severely overloaded. However, this application's embodiment calculates a weighted load value (integrating queue count, remaining time, average expected latency, overload degradation coefficient, and slow node correction coefficient), selects the optimal target chip node, and then matches the acceleration method with the highest historical success rate from the acceleration strategy library based on the target chip node's hardware parameters and the inference request characteristics (supporting automatic rollback when operators are missing), and estimates the completion time, achieving integrated collaborative decision-making for "balancing + acceleration".

[0067] Dynamic batch processing can improve throughput, but if GPU memory is not reserved before distribution, resource contention and failures are likely to occur under high concurrency. Therefore, this embodiment of the application also reserves resources before distributing inference tasks, groups tasks according to target chip nodes, pre-calculates and locks the required GPU memory, and reschedules if a task fails, in order to avoid resource conflict issues.

[0068] Step S4: Based on the optimal collaborative execution strategy, the inference task is sent to the corresponding target chip node and inference acceleration optimization is performed.

[0069] The optimal cooperative execution strategy is converted into corresponding execution instructions. These execution instructions contain specific parameters that the chip can execute, and are appended to the inference request.

[0070] For example, if the optimal co-execution strategy includes "model quantization", then a quantization bit width parameter (such as INT8 or 4-bit) is appended to the inference request. If the optimal co-execution strategy includes "operator fusion", then a fusion mode identifier parameter is appended to the inference request. If the optimal co-execution strategy includes "dynamic batching", then a batch size initial value parameter (default value is 4) is appended to the inference request. Other optimal co-execution strategies are appended with corresponding configuration parameters.

[0071] The execution instructions and the inference tasks are sent to the target chip node, and a batch processing buffer queue is generated locally on the target chip node. When the batch processing buffer queue meets any set condition, the inference tasks in the batch processing buffer queue are packaged into the execution queue of the target chip node, and inference acceleration optimization is performed according to the optimal acceleration scheme.

[0072] The set conditions include the number of inference tasks in the batch processing buffer queue reaching the current batch size, and the waiting time of the batch processing buffer queue exceeding a preset batch processing timeout threshold. The current batch size is determined by the current load state of the target chip node.

[0073] Optionally, the batch processing timeout threshold is set to 10ms.

[0074] Send the inference tasks that have completed resource reservation or have been packaged to the execution queue of the corresponding target chip node. At the same time, record the actual start timestamp of each inference task, calculate the estimated waiting time of each inference task, and write it to the system log.

[0075] The inference acceleration optimization is performed according to the aforementioned optimal acceleration scheme. The specific process is as follows: The corresponding operation is loaded based on the additional parameters. For example, based on the additional quantization bit width parameter, the quantization model version with the corresponding bit width is loaded and inference is executed. Based on the additional fusion mode identifier parameter, the pre-fused computation graph is used for execution, reducing GPU memory read / write operations. Then, based on the packaged inference task, batch inference is executed directly, and the results are split into single requests for return.

[0076] After each inference task is completed, the following information is obtained: request ID, target chip node ID, actual execution time (in milliseconds), task completion status, optimal acceleration scheme, peak memory usage, and precision sampling value. If the inference task fails, such as due to hardware error, timeout, or precision anomaly, the reason for failure is marked.

[0077] Finally, the execution logs of each inference task are integrated.

[0078] As an improvement to the appeal scheme, this application embodiment also introduces a continuous performance feedback and iterative optimization system for closed-loop optimization of the entire appeal task scheduling process. The system includes: (1) Performance data recording and aggregation: The system aggregates the execution logs by chip model, task type and acceleration strategy at fixed intervals (e.g. every 60 seconds or every 1000 tasks), and calculates the average time consumption, P99 time consumption, success rate and average accuracy loss of each group.

[0079] (2) Runtime Dynamic Calibration: During each aggregation cycle, the system also monitors the load fluctuations and anomalies of each chip node in real time, and directly uses the calibration results for subsequent requests within the same cycle. For example: If the actual latency of a chip node consistently exceeds twice the global average expected latency parameter, it will be marked as a "slow node" and its load balancing weight will be temporarily reduced.

[0080] If the execution queue length of a chip node exceeds the threshold, it is marked as "overloaded" and the weight is reduced to 0.2. At the same time, the chip node is notified to reduce the dynamic batch size or roll back the quantization level.

[0081] If a chip node experiences a hardware failure or consecutive timeouts, it will be immediately isolated, and the queued tasks will be migrated to other chip nodes.

[0082] The calibration results will be written to the chip status report in real time, so that the load balancing decision can take effect immediately within the current cycle.

[0083] (3) Strategy learning optimizer: By analyzing historical aggregated data, the system automatically adjusts the acceleration strategy library and load balancing weight baseline.

[0084] For chip models, task types, and acceleration strategy groups, if the average time consumption is lower than the preset lower limit threshold of the baseline value and the accuracy loss is less than 1%, the historical success count of that group will be increased; if the average time consumption increases by more than 30% of the baseline value or the success rate is lower than 90%, the success count will be reduced.

[0085] For each chip model and its corresponding task type, the average expected latency parameter is updated by sliding based on the long-term actual time consumption, and the initial weight benchmark for load balancing is recalculated (the lower the expected latency, the higher the weight). The updated parameters are used in step S3 in the next cycle.

[0086] Implementing the embodiments of this application has the following beneficial effects: This application embodiment collects real-time operating data from each chip node to obtain dynamic load information, and acquires its hardware parameters and average expected latency parameters to determine the actual load capacity of the chip node. The constructed real-time computing power status report provides a reliable basis for subsequent load balancing decisions. The inference task is structured and analyzed to obtain inference request features related to the computing power required by the inference task. This allows the system to allocate computing power resources to the inference task on demand, identify target chip nodes with sufficient computing power and the ability to execute inference tasks quickly, and improve the execution efficiency of inference tasks. Furthermore, by combining a preset acceleration strategy library with the inference request features, the system finds the most suitable acceleration strategy for the target chip node, avoiding situations where the selected chip, although compatible, is severely overloaded, leading to uneven load distribution among chips. Moreover, the strategy matching also considers task type and expected completion time, taking into account the compatibility between the chip and the inference task and the timeliness of task completion, achieving optimal matching of chip computing power resources and inference tasks, improving the overall execution efficiency and acceleration effect of inference tasks, while avoiding resource overload or idleness.

[0087] Second Embodiment Furthermore, in order to execute the chip computing power-based model inference task scheduling system corresponding to the above method embodiments, and to achieve the corresponding functions and technical effects, Figure 2 A specific structural diagram of a model inference task scheduling system based on chip computing power inference is provided. For ease of explanation, only the parts relevant to this embodiment are shown. The model inference task scheduling system based on chip computing power inference provided in this application embodiment includes: The chip computing power recording module 201 is used to construct a real-time computing power status report of the chip computing power cluster based on the real-time operating data, hardware parameters and average expected latency parameters of each chip node in the chip computing power cluster.

[0088] This application embodiment first obtains the real-time operating data of each chip node in the domestic chip computing power cluster, including the number of tasks to be executed at present, the estimated total remaining time of the tasks being executed, and the current available video memory of each chip node.

[0089] The hardware parameters are obtained by collecting the floating-point computing power, integer computing power, memory bandwidth, and supported operator list of each chip node.

[0090] Then, the historical inference task completion data of each chip node is executed through a preset sliding window, and the average time taken for different chip models to execute different types of inference tasks is calculated as the average expected delay parameter for each chip node to execute different types of inference tasks.

[0091] For example, for a speech recognition task (a type of inference task) and the chip node executing the task, 50 successful completion records related to speech recognition in the last 10 seconds are obtained. The actual time consumption data of this inference task are [5,6,5,7,5,…]. The average actual time consumption is calculated to be 5.8ms through the sliding window. Then, the average value is updated as the average expected latency parameter for the chip node to execute the speech recognition task.

[0092] Finally, the real-time running data, the hardware parameters, and the average expected latency parameters are associated with the corresponding chip nodes (associated through the chip node IDs) and stored in a preset structured database to obtain a real-time computing power status report of the chip computing power cluster.

[0093] Each data entry in the real-time computing power status report is indexed by the chip node ID, and the overload indicator field reflects the current load status of each chip node (the presence of the overload indicator field indicates a higher chip node load, while the absence of the overload indicator field indicates a lower chip node load).

[0094] Based on the real-time computing power status report, computing power data is provided to support subsequent decisions on maintaining chip load balance.

[0095] The inference request feature generation module 202 is used to generate corresponding inference request features for the inference task based on the preset business latency requirements and the data volume of the inference task; wherein, the inference request features include task type, computing power requirement level, expected completion time limit, recommended chip model and request ID.

[0096] The inference tasks received by domestic chip computing power clusters are mostly of three types: speech recognition, image classification, and large model generation. Therefore, the embodiments of this application focus on these three types of tasks.

[0097] First, the received inference task is parsed to determine which of the three task types mentioned above it belongs to. Then, based on the computation method corresponding to the task type, the data volume of the inference task is calculated. This data volume actually reflects the computational resources required to complete the inference task and is directly linked to the memory resources of the chip node.

[0098] Specifically, the `type` field in the inference task is parsed or matched using rules to determine the corresponding task type. For example, if the keyword "audio" is present, it's classified as speech recognition; if the keyword "image" is present, it's classified as image classification; and if the keyword "text" or "chat" is present, it's classified as large model generation. Based on the task type, the data volume of the inference task is determined. The data volume is calculated in three ways: (1) Calculation method for speech recognition: Based on the inference task, the audio duration and sampling rate are statistically analyzed, and the data volume is determined based on the product of the audio duration and sampling rate. The unit is uniformly recorded according to the original value. (2) Calculation method for image classification: According to the reasoning task, the image width, height and number of channels are counted. The number of channels for RGB images is fixed at 3. The data volume of a single image is determined based on the product of the image width, height and number of channels. The data volume of multiple images is the sum of the data volumes of each single image. (3) Calculation method for generating large models: Based on the inference task, the number of input text tokens and the number of expected output tokens are counted, and the total number of input text tokens and the number of expected output tokens are accumulated to determine the data volume.

[0099] Then, based on the data volume and the business latency requirements of the inference task, the computing power requirement level of the inference task is determined, and the maximum execution time of the computing power requirement level is taken as the expected completion time of the inference task.

[0100] Specifically, the latency requirement is first determined based on the business scenario corresponding to the inference task. If the data volume is small and the latency requirement is short, the corresponding computing power requirement level is higher to complete the inference task as quickly as possible. If the latency requirement is short but the data volume is large, it cannot be directly assumed to be a high computing power requirement level. Therefore, although the time limit is urgent, the large data volume makes it impossible to complete quickly, and forcing it to be at a high level would overwhelm the system. Therefore, the inference task is generally placed at a medium computing power requirement level, and the appropriate chip node is selected by subsequent load balancing weighted selection.

[0101] For example, this application provides three computing power requirement levels, as detailed below: High computing power requirement level: The maximum execution time is <100ms, and the data volume is relatively small; Medium computing power requirement level: The maximum execution time is between 100ms and 300ms, and the data volume is medium; Low computing power requirement level: The maximum execution time is >300ms and the data volume is large.

[0102] Furthermore, embodiments of this application will also consider the task type to find recommended chip models for the inference task. For example, low-power chip nodes are preferred for speech recognition tasks; general-purpose chip nodes are preferred for image classification tasks; and high-performance large-model chip nodes are preferred for large-model generation tasks.

[0103] Then, a unique request ID is generated for the inference task. The task type, computing power requirement level, expected completion time limit, recommended chip model and request ID are combined to generate corresponding inference request features for the inference task, so as to provide task-dimensional data support for subsequent task scheduling and matching.

[0104] The execution strategy generation module 203 is used to allocate corresponding target chip nodes to the inference task according to the real-time computing power status report, and generate the optimal collaborative execution strategy for the target chip node to execute the inference task based on the preset acceleration strategy library and the inference request characteristics.

[0105] In this step, the inference request characteristics of the inference task are read, and scheduling decisions are made in conjunction with the status of each chip node provided by the real-time computing power status report, so as to match the inference task with a suitable target chip node and the optimal collaborative execution strategy.

[0106] The received inference tasks are integrated into a task queue, and the following implementation steps S3.1 to S3.5 related to chip matching and acceleration strategy extraction are performed in sequence according to the priority corresponding to the computing power requirement level.

[0107] Step S3.1: Remove chip nodes that are in a fault state or isolated state from the real-time computing power status report, and remove chip nodes whose current available video memory is less than a first threshold. Generate a candidate node set based on the remaining chip nodes after removal. The fault state or isolated state reflects that the chip node is unavailable and cannot perform inference tasks, so it is removed.

[0108] Optionally, a first threshold is set as the minimum video memory required for the inference task (related to the data volume and the task type), for example, a large model generation task requires at least 8GB, and an image classification task requires 2GB.

[0109] Step S3.2: First, based on the real-time running data, calculate the weighted load value of the currently executing task by using the number of tasks to be executed at the candidate node, the estimated remaining total time of the currently executing task, and the preset dynamic weight.

[0110] The weighted load value L The calculation formula is: ; in, Q This represents the number of tasks currently pending execution for the candidate node. T The estimated total remaining time for the task currently in progress. W These are preset dynamic weights.

[0111] The dynamic weight W The calculation method is as follows: ; in, The average expected latency parameter for performing the current type of inference task on the candidate node. This is the overload degradation coefficient. If the candidate node has an overload identifier field, the value is 0.2; otherwise, the value is 1.0. This is the slow node correction coefficient. If the candidate node is marked as a slow node (the ratio of the actual time of a chip node to the average expected latency parameter under this task type exceeds the preset ratio threshold), the value is 0.5; otherwise, the value is 1.

[0112] The smaller the weighted load value, the more suitable the corresponding candidate node is for executing the inference request. Therefore, the candidate node with the smallest weighted load value is selected from the candidate node set as the target chip node for the inference task.

[0113] If multiple candidate nodes have the same and lowest weighted load values, the candidate node that best matches the task type is selected as the target chip node for the inference task. For example, a low-power chip node is preferred for a speech recognition task.

[0114] Step S3.3: Using the combination of the target chip node and the task type as the matching target, extract all acceleration candidate solutions that match the matching target from the acceleration strategy library. The acceleration strategy library stores the historical success rates of different acceleration candidate solutions under various combinations of chip nodes and task types.

[0115] Specifically, based on the matching target, all available acceleration candidate schemes are retrieved from the acceleration strategy library, and based on the target chip node's supported operator list, acceleration candidate schemes that are unavailable due to missing operators are eliminated. For example, if the target chip node's supported operator list lacks the BatchMatMul operator, but the INT8 inference task after model quantization requires this operator, then the acceleration candidate scheme for quantized INT8 is marked as unavailable.

[0116] Based on the expected completion time of the inference task, the current load status of the target chip node, and the hardware parameters, the eliminated acceleration candidate schemes are pre-filtered, and the acceleration candidate scheme with the lowest latency is selected first. If the target chip node is currently overloaded or is a slow node, a lightweight acceleration method is selected; if the data volume is extremely small, dynamic batch processing is skipped.

[0117] For the pre-filtered acceleration candidate solutions, the acceleration candidate solution with the highest historical success rate is directly selected as the optimal acceleration solution.

[0118] Furthermore, if the target chip node's list of supported operators is missing the operators required for the inference task, operator fusion is automatically disabled, and the acceleration candidate with the lowest software simulation overhead is selected; if the expected completion time is less than 1.5 times the average expected latency parameter, the acceleration candidate with the lowest latency is selected first.

[0119] Step S3.4: Calculate the estimated completion time of the reasoning task. The calculation formula is as follows: ; in, For the current time, The average expected latency parameter for performing the current inference task on the target chip node. This represents the number of tasks currently pending execution on the target chip node.

[0120] If the estimated completion time is compared with the expected completion time limit, and the result significantly exceeds the preset time limit threshold, an early warning flag is triggered for the inference request, and the inference request is given special attention.

[0121] Step S3.5: Based on the optimal acceleration scheme, the request ID, the target chip node ID, the estimated completion time of the inference task, and the warning flag (if triggered), generate the optimal collaborative execution strategy for the target chip node to execute the inference task.

[0122] As an improvement to the above solution, this embodiment of the application, before distributing the inference task to the target chip node, will also reserve computing resources for the inference task by detecting the memory status of the target chip node, specifically including the following steps: (1) Integrate the optimal collaborative execution strategies corresponding to each inference task into a collaborative execution strategy table. Group all inference tasks in the collaborative execution strategy table according to the ID of the target chip node. Inference tasks belonging to the same target chip node are assigned to the same group. Then, the inference tasks in the same group are grouped into corresponding queues to be distributed. (2) For each inference task in the distribution queue, perform the following reserved computing resource operations: Based on the data volume and task type of the inference task, calculate the GPU memory resource requirements of the inference task, including model weights, intermediate activation data, key-value cache, etc.

[0123] In the currently available video memory of the target chip node, the video memory value corresponding to the video memory resource requirement of the inference task is pre-locked, and the currently available video memory field in the real-time computing power status report is updated, that is, the video memory value reserved this time is subtracted from the currently available video memory. If it is found that the currently available video memory of the target chip node is insufficient during the above reservation process (for example, due to instantaneous resource contention caused by concurrent reservation, the remaining currently available video memory field after subtraction is negative), the inference request is marked as "resource reservation failed", and the corresponding target chip node is reallocated to the inference task from the candidate nodes.

[0124] In existing technologies, traditional load balancing and acceleration strategies are independent of each other, resulting in selected chips with low load but lacking support for the required operators, or selected chips that are compatible but severely overloaded. However, this application's embodiment calculates a weighted load value (integrating queue count, remaining time, average expected latency, overload degradation coefficient, and slow node correction coefficient), selects the optimal target chip node, and then matches the acceleration method with the highest historical success rate from the acceleration strategy library based on the target chip node's hardware parameters and the inference request characteristics (supporting automatic rollback when operators are missing), and estimates the completion time, achieving integrated collaborative decision-making for "balancing + acceleration".

[0125] Dynamic batch processing can improve throughput, but if GPU memory is not reserved before distribution, resource contention and failures are likely to occur under high concurrency. Therefore, this embodiment of the application also reserves resources before distributing inference tasks, groups tasks according to target chip nodes, pre-calculates and locks the required GPU memory, and reschedules if a task fails, in order to avoid resource conflict issues.

[0126] The inference task execution module 204 is used to send the inference task to the corresponding target chip node and perform inference acceleration optimization based on the optimal collaborative execution strategy.

[0127] The optimal cooperative execution strategy is converted into corresponding execution instructions. These execution instructions contain specific parameters that the chip can execute, and are appended to the inference request.

[0128] For example, if the optimal co-execution strategy includes "model quantization", then a quantization bit width parameter (such as INT8 or 4-bit) is appended to the inference request. If the optimal co-execution strategy includes "operator fusion", then a fusion mode identifier parameter is appended to the inference request. If the optimal co-execution strategy includes "dynamic batching", then a batch size initial value parameter (default value is 4) is appended to the inference request. Other optimal co-execution strategies are appended with corresponding configuration parameters.

[0129] The execution instructions and the inference tasks are sent to the target chip node, and a batch processing buffer queue is generated locally on the target chip node. When the batch processing buffer queue meets any set condition, the inference tasks in the batch processing buffer queue are packaged into the execution queue of the target chip node, and inference acceleration optimization is performed according to the optimal collaborative execution strategy.

[0130] The set conditions include the number of inference tasks in the batch processing buffer queue reaching the current batch size, and the waiting time of the batch processing buffer queue exceeding a preset batch processing timeout threshold. The current batch size is determined by the current load state of the target chip node.

[0131] Optionally, the batch processing timeout threshold is set to 10ms.

[0132] Send the inference tasks that have completed resource reservation or have been packaged to the execution queue of the corresponding target chip node. At the same time, record the actual start timestamp of each inference task, calculate the estimated waiting time of each inference task, and write it to the system log.

[0133] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, or improvements made by those skilled in the art within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for scheduling model inference tasks based on chip computing power inference, characterized in that, include: Based on the real-time operating data, hardware parameters, and average expected latency parameters of each chip node in the chip computing power cluster, a real-time computing power status report of the chip computing power cluster is constructed. Based on the preset business latency requirements and the data volume of the inference task, corresponding inference request features are generated for the inference task; wherein, the inference request features include task type, computing power requirement level, expected completion time limit, recommended chip model and request ID; Based on the real-time computing power status report, a corresponding target chip node is allocated to the inference task. Based on the preset acceleration strategy library and the inference request characteristics, the optimal collaborative execution strategy for the target chip node to execute the inference task is generated. Based on the optimal collaborative execution strategy, the inference task is sent to the corresponding target chip node and inference acceleration optimization is performed.

2. The model inference task scheduling method based on chip computing power inference according to claim 1, characterized in that, Based on the real-time operating data, hardware parameters, and average expected latency parameters of each chip node in the chip computing power cluster, a real-time computing power status report of the chip computing power cluster is constructed, specifically as follows: The real-time running data is obtained by collecting the number of tasks to be executed, the estimated total time remaining of tasks currently being executed, and the current available video memory for each chip node; The floating-point computing power, integer computing power, memory bandwidth, and list of supported operators of each chip node are collected to obtain the hardware parameters. The historical inference task completion data of each chip node is executed through a preset sliding window, and the average time taken for different types of inference tasks to be executed by different chip models is calculated to obtain the average expected latency parameter for each chip node to execute different types of inference tasks. The real-time operating data, the hardware parameters, and the average expected latency parameters are associated with the corresponding chip nodes and stored in a preset structured database to obtain the real-time computing power status report.

3. The model inference task scheduling method based on chip computing power inference according to claim 1, characterized in that, Based on the preset business latency requirements and the data volume of the inference task, corresponding inference request features are generated for the inference task, specifically: The received inference task is parsed to determine the task type, and the data volume of the inference task is calculated based on the calculation method corresponding to the task type. Based on the data volume and the business latency requirements of the inference task, the computing power requirement level of the inference task is determined, and the maximum execution time of the computing power requirement level is taken as the expected completion time of the inference task; wherein, the business latency requirements are determined by the business scenario corresponding to the inference task. Based on the task type, the recommended chip model for the inference task is determined.

4. The model inference task scheduling method based on chip computing power inference according to claim 1, characterized in that, Based on the real-time computing power status report, a corresponding target chip node is allocated to the inference task. Based on a preset acceleration strategy library and the inference request characteristics, an optimal collaborative execution strategy is generated for the target chip node to execute the inference task, specifically: Remove chip nodes that are in a faulty or isolated state from the real-time computing power status report, and remove chip nodes whose current available video memory is less than a first threshold. Generate a candidate node set based on the remaining chip nodes after removal. Based on the real-time operational data, a corresponding target chip node is allocated to the inference task from the candidate node set; Combining the hardware parameters of the target chip node and the characteristics of the inference request, the optimal acceleration scheme is matched for the inference task from the acceleration strategy library; The optimal collaborative execution strategy is generated based on the optimal acceleration scheme, the request ID, the target chip node ID, and the estimated completion time of the inference task.

5. The model inference task scheduling method based on chip computing power inference according to claim 4, characterized in that, Based on the real-time operational data, a corresponding target chip node is allocated to the inference task from the candidate node set, specifically as follows: Based on the real-time running data, the weighted load value of the currently executing tasks is calculated using the current number of tasks to be executed by the candidate node, the estimated total remaining time of the currently executing tasks, and the preset dynamic weights. The candidate node corresponding to the smallest weighted load value is selected as the target chip node for the inference task; If there are multiple minimum weighted load values, then the corresponding target chip node is selected for the inference task according to the task type.

6. The model inference task scheduling method based on chip computing power inference according to claim 4, characterized in that, Based on the hardware parameters of the target chip node and the characteristics of the inference request, the optimal acceleration scheme is matched for the inference task from the acceleration strategy library, specifically as follows: The combination of the target chip node and the task type is used as the matching target, and all acceleration candidate schemes that meet the matching target are extracted from the acceleration strategy library. Based on the hardware parameters and the expected completion time, the acceleration candidate solutions are screened, and then the acceleration candidate solution with the highest historical success rate is selected as the optimal acceleration solution from the screening results.

7. The model inference task scheduling method based on chip computing power inference according to claim 4, characterized in that, Also includes: Calculate the GPU memory resource requirements for the inference task based on the data volume and the task type. If the required video memory resources are greater than the currently available video memory of the target chip node, then a corresponding target chip node is reallocated for the inference task from the candidate nodes.

8. The model inference task scheduling method based on chip computing power inference according to claim 1, characterized in that, Based on the aforementioned optimal collaborative execution strategy, the inference task is sent to the corresponding target chip node and inference acceleration optimization is performed, specifically as follows: The optimal collaborative execution strategy is converted into corresponding execution instructions, and the execution instructions and the inference task are sent to the target chip node; A batch processing buffer queue is generated locally on the target chip node. When the batch processing buffer queue meets any set condition, the inference tasks in the batch processing buffer queue are packaged into the execution queue of the target chip node for inference acceleration optimization.

9. The model inference task scheduling method based on chip computing power inference according to claim 8, characterized in that, The set conditions include the number of inference tasks in the batch processing buffer queue reaching the current batch size, and the waiting time of the batch processing buffer queue exceeding a preset batch processing timeout threshold. The current batch size is determined by the current load status of the target chip node.

10. A model inference task scheduling system based on chip computing power inference, characterized in that, include: Chip computing power recording module, inference request feature generation module, execution strategy generation module, and inference task execution module; Among them, the chip computing power recording module is used to construct a real-time computing power status report of the chip computing power cluster based on the real-time operating data, hardware parameters and average expected latency parameters of each chip node in the chip computing power cluster. The inference request feature generation module is used to generate corresponding inference request features for inference tasks based on preset business latency requirements and the data volume of inference tasks; wherein, the inference request features include task type, computing power requirement level, expected completion time limit, recommended chip model and request ID; The execution strategy generation module is used to allocate corresponding target chip nodes to inference tasks based on the real-time computing power status report, and generate the optimal collaborative execution strategy for the target chip nodes to execute inference tasks based on the preset acceleration strategy library and the inference request characteristics. The inference task execution module is used to send the inference task to the corresponding target chip node and perform inference acceleration optimization based on the optimal collaborative execution strategy.

Images