A heterogeneous AI chip scheduling method based on unified runtime and performance prediction

Abstract

The application provides a heterogeneous AI chip scheduling method based on a unified runtime and performance prediction, belongs to the technical field of artificial intelligence, is applied to a computing cluster including multiple types of AI chips, acquires an AI task request containing source code, and parses the source code to generate a task feature vector representing task computing characteristics.The task feature vector and real-time state information of each AI chip in the computing cluster are jointly input into a trained performance prediction model to predict multiple execution indicators of the task on each chip, and a multi-objective optimization evaluation is performed in combination with execution priorities to determine an optimal target AI chip and generate a chip allocation decision containing a node and device identifier of the target AI chip.The source code is automatically converted into a unified hardware-independent intermediate representation, and then mapped and compiled into a native executable package for the determined target AI chip and positioned on the chip for deployment and execution according to the chip allocation decision, thereby improving resource scheduling efficiency.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a heterogeneous AI chip scheduling method based on unified runtime and performance prediction. Background Technology

[0002] In the field of artificial intelligence computing, to meet the ever-increasing demand for computing power and optimize costs, heterogeneous computing clusters composed of AI chips from various manufacturers are typically built. Therefore, researching intelligent scheduling methods that can automatically and efficiently allocate AI tasks to the most suitable chips for execution in such heterogeneous environments has significant application prospects for improving overall resource utilization and reducing operating costs.

[0003] Existing scheduling methods for heterogeneous environments, such as an AI chip adaptive deployment method based on dynamic heterogeneous resource awareness, mainly use a Long Short-Term Memory (LSTM) network model to predict the computing resources required for task requests and reserve resources on the target AI chip node based on the prediction results to achieve adaptive deployment of tasks.

[0004] However, the aforementioned existing technologies still have significant shortcomings in practical applications. On the one hand, the failure to resolve compatibility issues between the underlying software stacks of different chips means that AI task code written for one chip cannot run directly on another, resulting in high code porting and maintenance costs. On the other hand, predicting only a general computing resource requirement cannot accurately reflect the actual execution performance of tasks on different chip architectures. This leads to insufficient basis for scheduling decisions, making it difficult to achieve optimal resource allocation and thus limiting the improvement of overall cluster operating efficiency. Therefore, existing technologies suffer from low resource scheduling efficiency. Summary of the Invention

[0005] The purpose of this application is to provide a heterogeneous AI chip scheduling method, system, electronic device, and storage medium based on unified runtime and performance prediction, so as to solve the problem of low resource scheduling efficiency in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a heterogeneous AI chip scheduling method based on unified runtime and performance prediction, applicable to computing clusters including various types of AI chips. The method includes:

[0007] Obtain an AI task request that includes source code and real-time status information of each AI chip in the computing cluster. The AI ​​task request includes the execution priority.

[0008] The source code is parsed to generate computation graph data, and operator type and tensor shape information are extracted from the computation graph data to generate task feature vectors;

[0009] The task feature vector and the real-time status information of each AI chip are input into the trained performance prediction model to obtain multiple execution indicators for each AI chip to execute AI task requests, and a multi-dimensional performance prediction matrix is ​​generated based on the multiple execution indicators.

[0010] Based on the multidimensional performance prediction matrix and execution priority, a multi-objective optimization evaluation is performed to determine the target AI chip in the computing cluster and generate a chip allocation decision that includes the node and device identifiers of the target AI chip.

[0011] After converting the source code into a unified, hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to the native functions of the target AI chip and compiled to generate a native executable package. During the compilation process, the communication library calls of the distributed tasks are replaced with pre-defined unified communication interface calls.

[0012] Based on the node and device identifiers in the chip allocation decision, the target AI chip is located in the computing cluster, and a native executable package is deployed and executed on the target AI chip.

[0013] In one feasible implementation, the method further includes:

[0014] Obtain memory architecture data for each AI chip, including memory bandwidth and memory capacity;

[0015] By determining the tensor size and operator type in the computation graph data, the peak memory usage and computation-to-memory ratio of AI task requests are calculated.

[0016] A memory matching score is calculated for each AI chip by evaluating the fit between peak memory usage and memory capacity, and by calculating the matching degree between memory access ratio and memory bandwidth.

[0017] Based on the multidimensional performance prediction matrix and execution priority, a multi-objective optimization evaluation is performed to determine the target AI chip in the computing cluster, including:

[0018] Using memory matching score as a weighting factor, multi-objective optimization evaluation is performed based on multi-dimensional performance prediction matrix and execution priority to determine the target AI chip in the computing cluster.

[0019] In one feasible implementation, multi-objective optimization evaluation is performed using memory matching score as a weighting factor, based on a multi-dimensional performance prediction matrix and execution priority, to determine the target AI chip in the computing cluster, including:

[0020] The execution metrics of each AI chip in the multidimensional performance prediction matrix are normalized within the corresponding execution metric range of all AI chips to generate a normalized metric matrix.

[0021] The evaluation weights are assigned to each normalized index in the normalized index matrix according to the execution priority. The normalized indexes of each AI chip are multiplied by their corresponding evaluation weights and then summed to calculate the comprehensive evaluation score of each AI chip. The execution priority is at least one of the following: completion time priority, resource utilization priority, and energy consumption cost priority.

[0022] The overall evaluation score of each AI chip is multiplied by the memory matching score to obtain the scheduling score, and the AI ​​chip with the highest scheduling score is selected as the target AI chip.

[0023] In one feasible implementation, the method further includes:

[0024] By calculating the computational cost of each operator in the computation graph data, the performance-critical operators that account for the largest proportion of the total computational cost are identified.

[0025] For performance-critical operators, a variety of alternative operators adapted to the target AI chip are generated by systematically sampling within a search space consisting of tile size and memory access patterns.

[0026] Benchmarking multiple alternative operator implementations on the target AI chip to measure their respective execution times, and selecting the alternative operator implementation with the shortest execution time as the target operator implementation;

[0027] After converting the source code into a hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to native functions of the target AI chip and compiled to generate a native executable package, including:

[0028] After converting the source code into a unified, hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to the native functions of the target AI chip. During the compilation process, the standard function mapping for performance-critical operators is replaced by embedding the implementation of the target operator, ultimately generating a native executable package.

[0029] In one feasible implementation, after converting the source code into a unified, hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to native functions of the target AI chip. During compilation, the standard function mappings for performance-critical operators are replaced by embedding the target operator implementation, ultimately generating a native executable package, including:

[0030] By performing syntactic and semantic analysis on the source code to construct an abstract syntax tree, and generating a unified, hardware-independent intermediate representation based on the abstract syntax tree;

[0031] Traverse all function calls in the intermediate representation and identify the target function call corresponding to the performance key operator by comparing the name of each function call with the name of the performance key operator;

[0032] For target function calls, the compilation path that generates standard native function calls is replaced by directly embedding the instruction sequence of the target operator implementation;

[0033] For function calls in the intermediate representation other than the target function call, the standard runtime function to native function mapping is performed to generate standard native function calls;

[0034] The embedded instruction sequence and standard native function calls are linked together to generate a native executable package.

[0035] In one feasible implementation, the source code is parsed to generate computation graph data, and operator type and tensor shape information are extracted from the computation graph data to generate a task feature vector, including:

[0036] By tracing the execution flow of the source code on the virtual device, the source code is converted into a computation graph data consisting of operator nodes representing computational operations and tensor edges representing data flows;

[0037] Traverse all operator nodes and tensor edges in the computation graph data to count the number of different types of operator nodes and record the shape and size of each tensor edge;

[0038] Based on the number of different types of operator nodes and the shape and size of each tensor edge, the computational density and memory access pattern of the AI ​​task request are calculated, and a structured feature set is constructed by combining the computational density and memory access pattern, as well as the number of different types of operator nodes and the shape and size of each tensor edge.

[0039] The number of different types of operator nodes, the shape and size of each tensor edge, the computational density, and the memory access patterns in the structured feature set are transformed and concatenated using preset numerical encoding rules to generate task feature vectors.

[0040] In one feasible implementation, before inputting the task feature vector and the real-time state information of each AI chip into the trained performance prediction model to obtain the multidimensional performance prediction matrix, the method includes:

[0041] Obtain a training sample set, which includes multiple training samples. Each training sample includes a task feature vector generated from historical AI task requests, the real-time status information of the AI ​​chip at the time the historical AI task request was executed, and the corresponding real execution metrics.

[0042] For each training sample in the training sample set, perform the following steps:

[0043] The task feature vector and real-time status information of each training sample are input into a pre-defined performance prediction model to obtain the predicted performance index.

[0044] The loss function value of the performance prediction model is determined based on the actual and predicted performance metrics of each training sample.

[0045] If the loss function value does not meet the preset training stopping condition, adjust the model parameters of the performance prediction model to obtain an updated performance prediction model. Then, input the task feature vector and real-time state information of each training sample into a preset performance prediction model to obtain the predicted execution index until the loss function value meets the training stopping condition, and obtain the trained performance prediction model.

[0046] Secondly, this application provides a heterogeneous AI chip scheduling system based on unified runtime and performance prediction, applicable to computing clusters including various types of AI chips, including:

[0047] The acquisition module is used to acquire an AI task request that includes source code and real-time status information of each AI chip in the computing cluster. The AI ​​task request includes the execution priority.

[0048] The generation module is used to parse the source code to generate computation graph data and extract operator type and tensor shape information from the computation graph data to generate task feature vectors.

[0049] The prediction module is used to input the task feature vector and the real-time status information of each AI chip into the trained performance prediction model to obtain a multi-dimensional performance prediction matrix. The multi-dimensional performance prediction matrix includes multiple execution metrics for each AI chip to execute AI task requests.

[0050] The determination module is used to perform multi-objective optimization evaluation based on a multi-dimensional performance prediction matrix and execution priority to determine the target AI chip in the computing cluster and generate a chip allocation decision that includes the node and device identifiers of the target AI chip.

[0051] The generation module is also used to convert the source code into a unified hardware-independent intermediate representation, map the runtime functions in the intermediate representation to the native functions of the target AI chip, and compile them into a native executable package. During the compilation process, the communication library calls of the distributed task are replaced with the preset unified communication interface calls.

[0052] The deployment module is used to locate the target AI chip in the computing cluster based on the node and device identifiers in the chip allocation decision, and to deploy and execute the native executable package on the target AI chip.

[0053] Thirdly, this application provides an electronic device, comprising:

[0054] Memory, used to store computer programs;

[0055] A processor, used to execute computer programs, implements the steps of the heterogeneous AI chip scheduling method based on unified runtime and performance prediction as described in the first aspect above.

[0056] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the heterogeneous AI chip scheduling method based on unified runtime and performance prediction as described in the first aspect above.

[0057] The heterogeneous AI chip scheduling method based on unified runtime and performance prediction provided in this application first converts the user-submitted unmodified source code into a unified, hardware-independent intermediate representation, and then automatically compiles it into a native executable program for the target chip based on the scheduling results. This overcomes the compatibility barriers of the underlying software stacks between different chips and solves the technical problem in existing technologies where AI task code needs to be rewritten or adapted for different chips. Furthermore, by deeply analyzing the computation graph characteristics of the task and accurately predicting multi-dimensional execution performance indicators for each available AI chip in the cluster, it provides clear and quantitative data for scheduling decisions, solving the technical problem of insufficient decision-making basis due to inaccurate performance evaluation. Therefore, by deeply integrating the seamless execution capability of the code with accurate performance prediction decisions, all computing resources in the heterogeneous computing cluster can be included in a unified scheduling scope and optimally allocated, thereby significantly improving the overall resource scheduling efficiency.

[0058] Furthermore, by analyzing the memory access characteristics of AI tasks and calculating their matching degree with the physical memory architecture data of each heterogeneous AI chip, the basis for scheduling decisions is deepened from macroscopic performance prediction to the adaptation level between tasks and hardware micro-architecture. This allows the scheduler to clearly identify which chips' memory bandwidth or capacity better meets the needs of specific tasks, solving the technical problem of blind resource allocation caused by the inability to perceive the matching relationship between tasks and hardware memory characteristics in existing technologies. Therefore, by introducing the matching degree between tasks and chip memory architecture as a key decision factor, this application enables scheduling results to proactively avoid performance degradation caused by memory bottlenecks, thereby achieving more accurate and efficient resource allocation in heterogeneous environments. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A flowchart illustrating a heterogeneous AI chip scheduling method based on unified runtime and performance prediction, provided for an embodiment of this application;

[0061] Figure 2 A flowchart illustrating a method for generating task feature vectors provided in an embodiment of this application;

[0062] Figure 3 A schematic diagram of the structure of a heterogeneous AI chip scheduling system based on unified runtime and performance prediction provided in this application embodiment;

[0063] Figure 4 A schematic diagram of another heterogeneous AI chip scheduling system based on unified runtime and performance prediction provided in an embodiment of this application;

[0064] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] To address the problems of existing technologies, embodiments of this application provide a heterogeneous AI chip scheduling method, apparatus, device, computer storage medium, and computer program product based on unified runtime and performance prediction. The heterogeneous AI chip scheduling method based on unified runtime and performance prediction provided in this application embodiment will be described first below.

[0067] Figure 1 This illustration shows a flowchart of a heterogeneous AI chip scheduling method based on unified runtime and performance prediction, according to an embodiment of this application. The method is applicable to computing clusters that include various types of AI chips, such as… Figure 1 As shown, it includes:

[0068] A computing cluster refers to a system environment consisting of multiple computing nodes used to perform large-scale computing tasks. The various types of AI chips refer to the AI ​​chips from different vendors, with different hardware architectures and software ecosystems deployed on the computing nodes of the cluster. For example, type A chips and type B chips are incompatible with each other in terms of instruction sets, memory architectures or underlying drivers, thus forming a heterogeneous computing environment.

[0069] S110: Obtain an AI task request including source code and real-time status information of each AI chip in the computing cluster. The AI ​​task request includes the execution priority.

[0070] An AI task request is a data structure submitted by a user to the intelligent scheduling system to initiate an AI computing task. It includes source code describing the model logic and execution priorities for scheduling decisions. The source code is the user-written, unmodified program code defining the AI ​​model structure and training logic; it can be a script file based on a mainstream deep learning framework and using specific hardware acceleration functions. Real-time status information is dynamic data reflecting the current operating status of each AI chip in the computing cluster, periodically collected by the system from the monitoring interface of each chip. This can include chip computing unit utilization, memory usage, and current temperature, and can be a set of key-value pairs reported by device drivers or monitoring agents. Execution priorities are scheduling parameters set by the user in the AI ​​task request to guide the focus of multi-objective optimization evaluation; they can be one or a combination of priorities such as completion time priority, resource utilization priority, or energy cost priority.

[0071] In a cluster environment based on a containerized orchestration system, users initiate AI task requests by submitting a task description file that includes the source code path and execution priority definition. A task receiving component of the intelligent scheduling system listens for and receives this AI task request. Simultaneously, device plugins or monitoring agents deployed on each compute node periodically obtain real-time status information such as the compute unit utilization and memory usage of each AI chip by calling the AI ​​chip's driver interface, and aggregate this information to the intelligent scheduler. Ultimately, the intelligent scheduler integrates the AI ​​task request containing the source code and execution priority, along with the real-time status information of each AI chip in the cluster.

[0072] For example, in a multi-tenant AI cloud platform scenario, a user submits an AI task request to train an image classification model. The source code specified in the request is an address pointing to a code repository, and the execution priority is set to completion time first. At this time, the intelligent scheduler receives the request and simultaneously obtains the real-time status information of all currently available AI chip instances in the cluster from the monitoring system.

[0073] For example, there are four AI chip instances in the cluster: Chip Instance 1, Chip Instance 2, Chip Instance 3, and Chip Instance 4. Chip Instance 1 and Chip Instance 2 are both type A chips, but Chip Instance 1 has a computing unit utilization rate of 95% and its memory is nearing saturation, while Chip Instance 2's computing unit utilization rate is only 50%. Chip Instance 3 is a type B chip with a computing unit utilization rate of 10% and ample memory resources. Chip Instance 4 is a type C chip, currently completely idle with a computing unit utilization rate of 0%. Ultimately, the system will include the image classification model source code and AI task requests with completion time priority parameters, as well as the aforementioned specific real-time status information for Chip Instance 1, Chip Instance 2, Chip Instance 3, and Chip Instance 4.

[0074] S120: Parse the source code to generate computation graph data, and extract operator type and tensor shape information from the computation graph data to generate task feature vectors.

[0075] Computational graph data is a structured data representation parsed from source code to describe the computational flow of an AI model. It abstracts computational operations in the model as operator nodes and data flow between operations as data edges, forming a graphical structure that reflects computational logic and data dependencies. Operator type and tensor shape information are fundamental features extracted from computational graph data. Operator type refers to the specific type of computational operation that constitutes the operator node in the computational graph data, such as convolution or matrix multiplication. Tensor shape information describes the dimensionality of the multidimensional data array represented by the data edges in the computational graph data, such as the height, width, and number of channels of a four-dimensional array. The task feature vector is the final output of this step; it is a fixed-length numerical vector generated by numerically encoding the computational characteristics requested by the AI ​​task.

[0076] Based on the received AI task request containing source code, the execution flow of the source code is traced on a virtual device, or static syntax analysis is performed to parse the computational operations and data flow relationships defined in the code, thereby constructing a computational graph. Subsequently, the generated computational graph is traversed, the number of different operator types is counted, and the tensor shape information of key data edges is recorded. Finally, the counted number of operator types and the recorded tensor shape information are converted using a preset numerical encoding rule and concatenated into a fixed-dimensional numerical sequence to generate the final task feature vector for use in subsequent performance prediction models.

[0077] S130: Input the task feature vector and the real-time status information of each AI chip into the trained performance prediction model to obtain multiple execution indicators for each AI chip to execute AI task requests, and generate a multi-dimensional performance prediction matrix based on the multiple execution indicators.

[0078] The trained performance prediction model is a machine learning model trained on historical task execution data. This model can be a gradient boosting decision tree model capable of handling structured data or a neural network model capable of capturing sequential features. Multiple performance metrics refer to a set of predicted values ​​used to quantify the performance of a task on a specific chip. These values ​​can include estimated execution time, average power consumption, peak memory usage, or execution cost, etc. The multidimensional performance prediction matrix is ​​structured data generated in this step to summarize all prediction results. It is in the form of a two-dimensional table, where each row represents a specific AI chip instance, and each column represents a specific performance metric.

[0079] A performance prediction process is performed for each AI chip instance through a traversal process: its real-time state information is combined with the same task feature vector to form a complete input data for that chip. This input data is then fed into the trained performance prediction model for inference computation, thereby obtaining multiple performance metrics for the AI ​​task on that chip instance. Specifically, the task feature vector is combined with the real-time state information of each AI chip in the computing cluster to form multiple independent input samples, which are sequentially input into the trained performance prediction model to obtain multiple performance metrics for the AI ​​task on each chip. After repeating the above prediction process for all available AI chip instances in the cluster, each set of multiple performance metrics is summarized and organized as a row, ultimately generating a structured multidimensional performance prediction matrix. This matrix provides a comprehensive view of the expected performance of the task across all available hardware options.

[0080] For example, the task feature vector generated from the image classification model source code will now be processed together with the real-time state information of the four chip instances obtained in S110. First, for chip instance 1, its 95% utilization and high memory usage state information are combined with the task feature vector and input into the performance prediction model to obtain its multiple execution metrics [execution time: 300 seconds, execution cost: 1.5 units]. Next, the same operation is performed on chip instances 2, 3, and 4 respectively to obtain their respective multiple execution metrics. Finally, these four sets of execution metrics are summarized to generate the following multidimensional performance prediction matrix: Chip instance 1: [execution time: 300 seconds, execution cost: 1.5 units] Chip instance 2: [execution time: 180 seconds, execution cost: 0.9 units] Chip instance 3: [execution time: 150 seconds, execution cost: 0.8 units] Chip instance 4: [execution time: 120 seconds, execution cost: 0.7 units]. As can be seen from the multidimensional performance prediction matrix, although chip instance 1 and chip instance 2 are both type A chips, its predicted execution time (300 seconds) is much higher than that of chip instance 2 (180 seconds), which is of the same type but with a lighter load, due to its current real-time state of high computing unit utilization of 95% and near memory saturation. This demonstrates the accurate perception of the chip's real-time state by the performance prediction model.

[0081] S140: Based on the multidimensional performance prediction matrix and execution priority, perform multi-objective optimization evaluation to determine the target AI chip in the computing cluster and generate a chip allocation decision that includes the node and device identifiers of the target AI chip.

[0082] Multi-objective optimization evaluation is a decision-making process aimed at selecting a comprehensively optimal solution from multiple feasible options based on several potentially conflicting evaluation criteria. This process takes a multi-dimensional performance prediction matrix and execution priorities as input, and uses a specific evaluation algorithm to calculate a comprehensive score for each AI chip instance. Chip allocation decision is a data structure containing explicit instructions to tell subsequent steps which specific physical AI chip should be used. It includes node and device identifiers that uniquely identify the chip, such as the name of the computing node where the chip resides and the device number of the chip on that node.

[0083] First, the algorithm receives the multi-dimensional performance prediction matrix generated in the previous step and the execution priority set in the AI ​​task request. Next, a multi-objective decision-making algorithm, such as a weighted summation method, is initiated. This algorithm iterates through each row of the multi-dimensional performance prediction matrix, representing each AI chip instance. For each chip instance, the algorithm assigns different importance weights to multiple execution metrics based on the execution priority; for example, if the execution priority is completion time first, then the execution time metric will be given the highest weight. Then, the various execution metrics of each chip instance are calculated along with their corresponding weights to obtain a comprehensive evaluation score. After calculating the comprehensive evaluation scores for all chip instances, all scores are compared, and the AI ​​chip with the highest scheduling score is selected as the target AI chip. Finally, a chip allocation decision is made using the node and device identifier of this target AI chip.

[0084] S150: After converting the source code into a unified, hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to the native functions of the target AI chip and compiled to generate a native executable package. During the compilation process, the communication library calls of distributed tasks are replaced with preset unified communication interface calls.

[0085] Intermediate representation is a standardized code format independent of any specific hardware architecture. It acts as a bridge between source code and target machine code, carrying the pure computational logic parsed from the source code. Runtime functions refer to function calls provided by upper-level frameworks at the source code level to perform specific computations or manage hardware resources, such as a function to perform convolution operations. Native functions are low-level functions provided by the official software development kit (SDK) of the target AI chip that can be executed directly on that chip. A native executable package is a collection of binary code and related configurations that can run directly on the target AI chip. Communication library calls refer to functions in the source code used to implement data synchronization and exchange between multiple chips, and are usually tied to the ecosystem of a specific hardware vendor.

[0086] Unified Communication Interface (UCI) calls are standardized cross-vendor communication function calls designed to provide unified and efficient communication capabilities in heterogeneous environments. The pre-defined UCI is a set of abstract, hardware vendor-independent communication primitives, such as point-to-point communication, global broadcast, and global collection. It is implemented at the underlying level through a pluggable driver architecture. During compilation or runtime, based on the target AI chip type determined in the chip allocation decision, it dynamically binds to and calls the specific implementation of the optimal vendor's communication library (e.g., NCCL, HCCL, ACL, etc.) for that chip. This allows the upper-layer AI task source code to fully leverage the high-efficiency communication performance of the underlying hardware while maintaining interface consistency, without requiring modifications to the communication code for different chips.

[0087] First, the chip allocation decision specifying the target AI chip, generated in the previous step, is received. Based on this decision, a backend compilation process tailored to the target AI chip type is activated. This process begins by using a frontend compiler to parse and convert the user-provided source code into a unified intermediate representation. This involves performing syntactic and semantic analysis on the source code to construct an abstract syntax tree (AST), and then generating a unified, hardware-independent intermediate representation based on the AST. Next, this intermediate representation is traversed, and for each runtime function, it is replaced with the native function corresponding to the target AI chip according to a preset mapping rule. Specifically, if a communication library call for distributed training is identified during this process, it will be replaced with a preset unified communication interface call. After all functions have been mapped and replaced, the backend compiler for the target AI chip is invoked to compile and link the modified intermediate representation, ultimately generating a native executable package that can run directly on the target AI chip.

[0088] S160: Based on the node and device identifiers in the chip allocation decision, locate the target AI chip in the computing cluster, and deploy and execute the native executable package on the target AI chip.

[0089] A scheduling and execution component parses the node and device identifiers in the chip allocation decision. Based on the node identifier, this component issues instructions to the corresponding compute node in the computing cluster, requesting the creation of an isolated environment, such as a container, on that node for executing AI tasks. During the creation of this isolated environment, the device interface of the target AI chip is mounted to this environment based on the device identifier, ensuring that the task can only access the designated physical chip. Once the environment is ready, the native executable package generated in the previous step is loaded into this environment, and the execution command is finally triggered to start the execution of the native executable package. This completes the entire intelligent scheduling and deployment process.

[0090] For example, in S140, after multi-objective optimization evaluation, chip instance 4 is ultimately determined as the target AI chip for this image classification task, and a chip allocation decision containing its node and device identifier is generated. Now, in S160, the scheduling execution component receives this chip allocation decision and the native executable package compiled for the C-type chip. Based on the node identifier in the chip allocation decision, this component instructs the computing node where chip instance 4 resides to create a new container. When creating the container, the physical device interface of chip instance 4 is precisely mounted into the container according to the device identifier. Subsequently, the native executable package is deployed to the container's file system and execution is initiated. Finally, the user's image classification model begins to run efficiently on chip instance 4, which is currently the optimal chip selected by the scheduling system.

[0091] In one feasible implementation, the method further includes:

[0092] Obtain memory architecture data for each AI chip, including memory bandwidth and memory capacity.

[0093] Memory architecture data refers to a set of static parameters describing the core characteristics of an AI chip's physical memory system. Among these, memory bandwidth refers to the maximum data transfer rate the chip's memory system can achieve per unit of time, typically measured in gigabytes per second. Memory capacity refers to the total size of high-speed memory on the chip that can be directly used by AI tasks, also typically measured in gigabytes.

[0094] When the system initializes or a new type of AI chip is added to the cluster, the memory bandwidth and memory capacity of each AI chip are obtained by querying the chip's technical specification manual or using hardware information detection tools. These values ​​are then associated with the chip type or instance ID to form a memory architecture database. For example, the system records that type A chips have a memory capacity of 16 gigabytes and a memory bandwidth of 500 gigabytes per second; type B chips have a memory capacity of 32 gigabytes and a memory bandwidth of 900 gigabytes per second.

[0095] By determining the tensor size and operator type in the computation graph data, the peak memory usage and computation-to-memory ratio of AI task requests are calculated.

[0096] Peak memory usage refers to the maximum memory space required by an AI task throughout its entire execution lifecycle. This value determines whether the task can run successfully on a chip with a specific memory capacity. The compute-to-memory ratio (CTR) is a metric used to measure computational intensity. It represents the ratio between the total number of computational operations and the total amount of data accessed in memory during task execution. A high CTR indicates a compute-intensive task, while a low CTR indicates a memory-intensive task.

[0097] The size of each tensor is obtained by traversing all data edges in the computational graph data generated in S120, and the lifecycle of the tensors is analyzed in conjunction with the dependencies between operators, thereby estimating the peak memory usage during the entire task execution process. Simultaneously, by traversing all operator nodes, the required number of computational operations for each operator is estimated based on its type, accumulated, and then divided by the sum of all tensor sizes, thus calculating the overall computation-to-memory ratio of the task. For example, for an image classification model, analysis shows that its peak memory usage is 18 gigabytes, indicating a low computation-to-memory ratio, suggesting that it is a task sensitive to both memory capacity and bandwidth.

[0098] A memory matching score is calculated for each AI chip by evaluating the fit between peak memory usage and memory capacity, and by calculating the matching degree between memory access ratio and memory bandwidth.

[0099] The memory compatibility score is a numerical value used to quantify the compatibility between an AI task and the memory system of a specific AI chip. This score combines considerations of whether the memory capacity is sufficient and whether the bandwidth is matched. A high score indicates a lower risk of performance loss due to memory bottlenecks on that chip.

[0100] First, a capacity suitability assessment is performed. The peak memory usage of the task calculated in the previous step is compared with the memory capacity of each AI chip. If the chip's memory capacity is less than the task's requirements, the chip's memory matching score is a specific value indicating unavailability, such as 0. For chips that pass the capacity assessment, a bandwidth matching assessment is then performed. The computation-to-memory ratio of the task is compared with the chip's memory bandwidth. For tasks with low computation-to-memory ratios (i.e., memory-intensive tasks), chips with higher memory bandwidth will obtain higher matching scores. Continuing with the previous example, the peak memory usage of the task is 18 gigabytes. The memory capacity of type A chip is only 16 gigabytes, which does not meet the requirements, so its memory matching score is 0. Type B chip has a memory capacity of 32 gigabytes, which meets the requirements, and its higher memory bandwidth matches the low computation-to-memory ratio characteristic of the task, thus obtaining a higher memory matching score.

[0101] Step S140 performs multi-objective optimization evaluation based on the multi-dimensional performance prediction matrix and execution priority to determine the target AI chip in the computing cluster, including:

[0102] Using memory matching score as a weighting factor, multi-objective optimization evaluation is performed based on multi-dimensional performance prediction matrix and execution priority to determine the target AI chip in the computing cluster.

[0103] Using the memory matching score as a weighting factor means that in the multi-objective optimization evaluation process of S140, the memory matching score calculated in this embodiment is used as a multiplier or weighting term to adjust or correct the preliminary evaluation results based on performance prediction. After calculating the comprehensive evaluation score for each AI chip in S140, it is not directly compared. Instead, the comprehensive evaluation score of each chip is first combined with its corresponding memory matching score, for example, by performing a multiplication operation, to obtain a final scheduling score. This process ensures that the final score of chips that are better matched to the task's memory characteristics will be improved, while that of chips that are not will be suppressed. Finally, by comparing the final scheduling scores of all chips, the optimal target AI chip is determined.

[0104] This embodiment analyzes the memory access characteristics of AI tasks and calculates their matching degree with the physical memory architecture data of each heterogeneous AI chip. This deepens the scheduling decision-making basis from macroscopic performance prediction to the adaptation level between the task and the hardware micro-architecture. This allows the scheduler to clearly identify which chips' memory bandwidth or capacity better meets the needs of specific tasks, solving the technical problem of blind resource allocation caused by the inability to perceive the matching relationship between tasks and hardware memory characteristics in existing technologies. Therefore, by introducing the matching degree between tasks and chip memory architecture as a key decision factor, this application enables scheduling results to proactively avoid performance degradation caused by memory bottlenecks, thereby achieving more accurate and efficient resource allocation in heterogeneous environments.

[0105] In one feasible implementation, multi-objective optimization evaluation is performed using memory matching score as a weighting factor, based on a multi-dimensional performance prediction matrix and execution priority, to determine the target AI chip in the computing cluster, including:

[0106] The execution metrics of each AI chip in the multidimensional performance prediction matrix are normalized within the corresponding execution metric range of all AI chips to generate a normalized metric matrix.

[0107] The normalized index matrix is ​​a new matrix obtained by dimensionless processing of the original forecast data. Its purpose is to eliminate the numerical differences caused by different dimensions of different performance indicators, so that the indicators can be compared and weighted on a unified scale.

[0108] For each column of the multidimensional performance prediction matrix, i.e., each execution metric, a method such as min-max normalization is used. For metrics like execution time and execution cost, where smaller values ​​are better, a specific formula is used to convert them into a value between 0 and 1, where the original minimum value corresponds to the normalized 1 and the original maximum value corresponds to the normalized 0. For metrics like resource utilization, where larger values ​​are better, the original value is directly linearly mapped to the interval between 0 and 1. After normalizing all execution metric columns, a normalized metric matrix is ​​generated. For example, for the execution time of the four chip instances in the aforementioned embodiment, after normalization, a new set of dimensionless values ​​[0, 0.67, 0.83, 1.0] may be obtained.

[0109] The evaluation weights are assigned to each normalized index in the normalized index matrix according to the execution priority. The normalized indexes of each AI chip are multiplied by their corresponding evaluation weights and then summed to calculate the comprehensive evaluation score of each AI chip. The execution priority is at least one of the following: completion time priority, resource utilization priority, and energy consumption cost priority.

[0110] Evaluation weights are a set of values ​​representing the importance of each normalized metric in the final decision. These weights are directly influenced by the user-specified execution priorities. The comprehensive evaluation score is a preliminary score that integrates all performance prediction metrics, reflecting the overall performance of an AI chip in executing a task without considering memory matching. Execution priorities, such as completion time priority, resource utilization priority, and energy cost priority, mean that users can choose one or more of these three preset optimization objectives when submitting a task, indicating which performance metrics(s) should be emphasized in the multi-objective optimization evaluation process.

[0111] First, a set of evaluation weights is determined based on the execution priority set in the AI ​​task request. For example, if the execution priority is completion time first, a higher evaluation weight, such as 0.7, is assigned to the normalized execution time metric, while lower weights, such as 0.3, are assigned to other metrics. Then, each row of the normalized metric matrix is ​​iterated, and each normalized metric of each AI chip in that row is multiplied by its corresponding evaluation weight. All products are then summed to calculate the comprehensive evaluation score of the AI ​​chip. Continuing with the previous example, since the execution priority is completion time first, after weighted summation, the comprehensive evaluation scores of the four chip instances can be obtained, for example, 0.2, 0.6, 0.8, and 1.0 respectively.

[0112] The overall evaluation score of each AI chip is multiplied by the memory matching score to obtain the scheduling score, and the AI ​​chip with the highest scheduling score is selected as the target AI chip.

[0113] The scheduling score is the decisive score used for the final decision in this implementation, integrating both the macroscopic performance prediction of the AI ​​chip and the microscopic memory architecture matching degree. The comprehensive evaluation score calculated for each AI chip in the previous step is multiplied by the memory matching degree score calculated for that chip to obtain the final scheduling score for that chip. After calculating the scheduling scores for all available AI chip instances, all scores are compared, and the AI ​​chip with the highest scheduling score is determined as the target AI chip for this scheduling. Assume that the memory matching degree scores of chip instances 1 and 2 are 0, and the memory matching degree scores of chip instances 3 and 4 are 0.9 and 1.0, respectively. After multiplying the comprehensive evaluation score by the memory matching degree score, the final scheduling scores obtained are: 0 for chip instance 1, 0 for chip instance 2, 0.72 for chip instance 3, and 1.0 for chip instance 4. By comparison, chip instance 4 has the highest scheduling score, therefore, chip instance 4 is ultimately determined as the target AI chip.

[0114] In one feasible implementation, the method further includes:

[0115] By calculating the computational cost of each operator in the computation graph data, the performance-critical operators that account for the largest proportion of the total computational cost are identified.

[0116] Performance-critical operators refer to one or more computational operations that consume the most computational resources and have the greatest impact on the overall execution time in the entire computational process of an AI task. Computational cost is a quantitative assessment of the amount of computation required to execute an operator, usually expressed as the number of floating-point operations required.

[0117] First, each operator node in the computational graph data generated in S120 is traversed. Based on the type of each operator node and the shape and size of its input tensor, the computational cost required to execute that operator is estimated. After calculating the computational cost for each operator, they are summed to obtain the total computational cost. Finally, the proportion of each operator's computational cost in the total computational cost is calculated, and the operator with the highest proportion is identified as the performance-critical operator. For example, for an image classification model containing multiple operators such as convolution, activation, and pooling, calculations show that a specific convolution operator accounts for 60% of the total computational cost; therefore, this convolution operator is identified as the performance-critical operator.

[0118] For performance-critical operators, a variety of alternative operator implementations adapted to the target AI chip are generated by systematically sampling within a search space consisting of tile size and memory access patterns.

[0119] Alternate operator implementations refer to multiple code versions with the same functionality but potentially different performance, generated by adjusting their underlying code implementation for the same mathematical operation. The search space is the set of all possible values ​​for these adjustable implementation parameters. Systematic sampling is a strategy that selects a representative subset of parameter combinations from this space to avoid the enormous overhead of traversing all combinations.

[0120] First, based on the performance-critical operators identified in the previous step and the hardware characteristics of the target AI chip determined by S140, a search space for implementation parameters is defined. Then, a set of parameter combinations is selected from this space using a sampling strategy, and a specific candidate operator implementation code is generated for each selected parameter combination. Continuing the previous example, for the identified convolution operator, a search space consisting of different convolution algorithms (e.g., Img2Col or Winograd) and different data block tiling sizes can be defined, and three different candidate operator implementations can be generated from it, named Implementation A, Implementation B, and Implementation C, respectively.

[0121] Benchmarking was performed on multiple alternative operator implementations on the target AI chip to measure their respective execution times, and the alternative operator implementation with the shortest execution time was selected as the target operator implementation.

[0122] Benchmarking is the process of running code snippets on real hardware to precisely measure their performance metrics. The target operator implementation is the code version that, among all alternative versions, has been proven by benchmarking to perform optimally on the target AI chip.

[0123] All candidate operator implementations generated in the previous step are compiled and executed one by one on the target AI chip determined in S140, and the precise time required for each candidate operator implementation to complete the operation is measured. After all candidate versions have been tested, the recorded execution times are compared, and the version with the shortest execution time is determined as the final target operator implementation. Continuing the previous example, benchmark tests are performed on implementations A, B, and C on the target AI chip, and the measured execution times are 15 milliseconds, 8 milliseconds, and 11 milliseconds, respectively. Since implementation B has the shortest execution time, implementation B is selected as the target operator implementation for this scheduling.

[0124] Step S150 involves converting the source code into a hardware-independent intermediate representation, mapping the runtime functions in the intermediate representation to native functions of the target AI chip, and compiling to generate a native executable package, including:

[0125] After converting the source code into a hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to the native functions of the target AI chip. During the compilation process, the standard function mappings for performance-critical operators are replaced by embedding the implementation of the target operators, ultimately generating a native executable package.

[0126] Replacing standard function mapping by embedding target operator implementations means that in the S150 compilation process, when a call to a performance-critical operator is encountered, the general function mapping is no longer executed. Instead, the code of the target operator implementation with the best performance found in this embodiment is directly embedded into the final program.

[0127] The S150 compilation process has been modified into a smart compilation process with conditional checks. For most ordinary function calls in the intermediate representation, the process is consistent with the standard S150 description, performing a general function-to-native function mapping. However, when the compilation process encounters a call to a performance-critical operator, a special path is triggered: the standard mapping is abandoned, and instead, the code instructions of the target operator implementation determined in this implementation are directly inlined or linked to the current compilation location. This process ensures that the most computationally intensive parts of the AI ​​task can be executed on the target AI chip in the most efficient way.

[0128] This embodiment solves the performance loss problem that may be caused by general cross-platform compilation by automatically identifying the most time-consuming performance-critical operators in the AI ​​task during the compilation stage and searching for and replacing them with the best-performing underlying implementations on the target AI chip. This deep optimization of the critical path enables the final executable program to achieve near-theoretical peak computational performance on heterogeneous hardware, thereby significantly improving the execution efficiency of the task.

[0129] In one feasible implementation, after converting the source code into a unified, hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to native functions of the target AI chip. During compilation, the standard function mappings for performance-critical operators are replaced by embedding the target operator implementation, ultimately generating a native executable package, including:

[0130] An abstract syntax tree is constructed by performing syntactic and semantic analysis on the source code, and a unified, hardware-independent intermediate representation is generated based on the abstract syntax tree.

[0131] Abstract syntax trees (ASTs) are data structures that represent the syntactic structure of source code in a tree-like format. They ignore non-critical information such as punctuation marks in the code and only retain the core structure that reflects the program logic.

[0132] First, a front-end compiler performs lexical and syntactic analysis on the user-provided source code, converting it into an abstract syntax tree (AST). For example, a line of code defining a convolutional layer generates an AST node representing that function call, containing semantic information such as the input, output, and parameters of the convolution operation. Next, a conversion program traverses this AST, translating each node into a standardized, hardware-independent intermediate representation instruction according to predefined rules. For instance, the AST node representing the convolutional layer definition is translated into a unified intermediate representation instruction, such as an IR_CONV2D instruction that only defines the input data, weight data, and mathematical properties of the convolution operation; this instruction itself does not contain any hardware vendor-specific library information. After traversing and translating the entire AST, a unified, hardware-independent intermediate representation is generated.

[0133] Iterate through all function calls in the intermediate representation and identify the target function call corresponding to the performance key operator by comparing the name of each function call with the name of the performance key operator.

[0134] The target function call refers to the specific function call instance in the intermediate representation that is functionally completely corresponding to the performance key operator determined in the preceding steps.

[0135] A code scanning program examines each function call in the intermediate representation generated in S150 one by one. This program performs a string matching between the name of the currently examined function call and the name of the performance-critical operator identified in the previous step. If the names match, the function call is marked as the target function call. For example, when traversing the intermediate representation of the image classification model, the convolution function call corresponding to the performance-critical operator is identified through name matching and marked as the target function call.

[0136] For target function calls, the compilation path that generates standard native function calls is replaced by directly embedding the instruction sequence of the target operator implementation.

[0137] An instruction sequence refers to a low-level machine code that can be directly executed by the target AI chip after being compiled from the target operator implementation. Replacing the compilation path means interrupting the default, general code generation process during the compiler's code generation phase and instead using a specified, pre-optimized code block.

[0138] When the compiler encounters the target function call marked in the previous step, it triggers a special compiler directive. This directive prevents the compiler from generating standard, generic native function code for this function call. Instead, it directly copies and inserts the instruction sequence of the target operator implementation determined in the preceding steps into the currently generated program code. For example, when the compiler encounters this marked convolution function call, it no longer generates standard convolution code, but instead directly copies and inserts the instruction sequence of the target operator implementation B selected in the previous step with the shortest execution time into the program code.

[0139] For function calls in the intermediate representation other than the target function call, a standard runtime function-to-native function mapping is performed to generate a standard native function call.

[0140] Standard native function calls refer to the native function code generated through general mapping rules that guarantee correct functionality. For all other function calls not marked as target function calls, the compiler processes them according to the standard procedure described in S150. That is, based on the pre-defined mapping table, it finds and generates the corresponding native function call code on the target AI chip for each runtime function. This process ensures that all other functions in the entire program, except for performance-critical operators, can be correctly translated. For example, for other non-critical calls such as activation functions and pooling functions in the model, the compiler continues to execute standard function mapping to generate their standard native function calls on C-type chips.

[0141] The embedded instruction sequence and standard native function calls are linked together to generate a native executable package.

[0142] The linker performs address relocation and symbol resolution on the optimized instruction sequence for performance-critical operators embedded via a special path, along with all standard native function calls generated via the standard path, and other library functions required by the program. After these operations, a complete native executable package containing all optimizations and standard code is generated. For example, the linker ultimately combines the embedded optimized convolution instruction sequence with all standard native function calls to generate a high-performance native executable package tailored to chip instance 4.

[0143] Figure 2 A flowchart illustrating a method for generating task feature vectors according to an embodiment of this application is shown.

[0144] In one feasible implementation, step S120 parses the source code to generate computation graph data, and extracts operator type and tensor shape information from the computation graph data to generate task feature vectors, such as... Figure 2 As shown, the method includes:

[0145] S210: By tracing the execution flow of the source code on the virtual device, the source code is converted into a computation graph data consisting of operator nodes representing computational operations and tensor edges representing data flows.

[0146] An operator node is a basic unit in computation graph data, used to abstractly represent a specific computational operation, such as a convolution operation or a matrix multiplication. A tensor edge is a directed edge in computation graph data that connects two operator nodes, used to represent a multidimensional data array flowing between these two computational operations.

[0147] A code tracing tool is used to symbolically execute user-provided source code on a virtual device that does not perform actual numerical computations. During the tracing process, whenever a computational function call from a deep learning framework is encountered, a corresponding operator node is created in the computation graph data. Simultaneously, the input and output data of the function call are recorded, and tensor edges are created between the corresponding operator nodes to represent this data dependency. After tracing the entire source code, a complete computation graph data consisting of operator nodes and tensor edges is constructed. For example, for the aforementioned image classification model training task, tracing its source code generates a computation graph data containing multiple convolution, activation, pooling, and other operator nodes and corresponding tensor edges.

[0148] S220: Traverse all operator nodes and tensor edges in the computation graph data to count the number of different types of operator nodes and record the shape and size of each tensor edge.

[0149] A graph traversal algorithm, such as depth-first search, is used to access each operator node and tensor edge in the computational graph data generated in S210. When accessing an operator node, its type information is read and incremented for the corresponding item in a counter set used to count the number of each type of operator. When accessing a tensor edge, the dimension information, i.e., the shape and size of the data tensor it represents, is read and recorded. For example, traversing the computational graph data of this image classification model can identify that it contains 50 convolutional nodes, 50 activation nodes, etc., and record the shape and size of all tensor edges.

[0150] S230: Based on the number of different types of operator nodes and the shape and size of each tensor edge, calculate the computational density and memory access pattern of the AI ​​task request, and construct a structured feature set using the computational density, memory access pattern, number of different types of operator nodes, and shape and size of each tensor edge.

[0151] A structured feature set is a data structure used to aggregate all extracted and computed AI task features. It contains data in various forms, including numerical, list, and categorical data. First, based on the number of different types of operator nodes and the shape and size of each tensor edge as determined in the previous step, two deeper derived features are calculated. The computational intensity of the task is calculated by estimating the total computational cost of all operator nodes and dividing it by the total data volume of all tensor edges. The locality characteristics of its memory access patterns are determined by analyzing the topological structure of the computation graph data, such as the graph's depth, width, and branching. Finally, these two calculated derived features are stored together with the original features determined in the previous step in a structured feature set. For example, based on information such as the 50 convolutional nodes, the computational intensity of the model can be calculated to be a high value, indicating that it is a computationally intensive task, while the analysis also shows that its memory access patterns exhibit good locality.

[0152] S240: The number of different types of operator nodes, the shape and size of each tensor edge, the computational density, and the memory access pattern in the structured feature set are transformed and concatenated using preset numerical encoding rules to generate the task feature vector.

[0153] Numerical encoding rules are a set of predefined methods used to convert non-numerical or variable-length features into fixed-length numerical representations to meet the input requirements of subsequent machine learning models.

[0154] For each piece of information in the structured feature set constructed in the previous step, the corresponding numerical encoding rules are applied. For example, for numerical information such as the number of operator nodes, their original values ​​are used directly; for variable-length list information such as the shape and size of tensor edges, their maximum, minimum, and average values ​​are calculated to convert them into a set of fixed-number values. After all the information is converted into numerical form, they are concatenated in a preset order to generate a fixed-dimensional task feature vector. For example, all the information in the structured feature set is numerically encoded and concatenated. For instance, the 50 convolution nodes, the calculated computational density of 25.8, and the memory access pattern encoding value of 1 representing good locality, as shown in the previous example, are concatenated into a numerical vector of the form [50, 25.8, 1], which serves as the task feature vector for this image classification model.

[0155] In one feasible implementation, before step S130, which inputs the task feature vector and the real-time state information of each AI chip into the trained performance prediction model to obtain the multidimensional performance prediction matrix, the method includes:

[0156] Obtain a training sample set, which includes multiple training samples. Each training sample includes a task feature vector generated from historical AI task requests, the real-time status information of the AI ​​chip at the time the historical AI task request was executed, and the corresponding real execution metrics.

[0157] The training sample set refers to a dataset consisting of a large number of historical task execution records, used to train the performance prediction model. Each training sample is an independent data unit in this set, including the input features of a historical task and its final true performance result. True execution metrics refer to the performance data actually measured and recorded by the monitoring system after the historical AI task request execution is completed, such as the actual end-to-end runtime and average power consumption of the task. It serves as a label or ground truth in supervised learning.

[0158] For each training sample in the training sample set, perform the following steps: Input the task feature vector and real-time state information of each training sample into a pre-defined performance prediction model to obtain the predicted execution metric. Determine the loss function value of the performance prediction model based on the actual and predicted execution metrics of each training sample. If the loss function value does not meet the pre-defined training stopping condition, adjust the model parameters of the performance prediction model to obtain an updated performance prediction model, and return to the previous step of inputting the task feature vector and real-time state information of each training sample into a pre-defined performance prediction model to obtain the predicted execution metric, until the loss function value meets the training stopping condition, thus obtaining the trained performance prediction model.

[0159] First, a pre-defined performance prediction model is initialized. Then, a loop iteratively processes each training sample in the training sample set. In each iteration, the feature vector and state information of the current training sample are input into the model to obtain the predicted performance metric. Then, a loss function, such as the mean squared error function, is used to calculate the loss function value between the predicted metric and the true metric in the sample. Finally, based on this loss function value, an optimization algorithm, such as gradient descent, is used to fine-tune the model's internal parameters. Training stops when the entire training sample set has been processed a predetermined number of times, or when the overall prediction loss of the model falls below a set threshold, at which point the training is complete, resulting in the trained performance prediction model.

[0160] The heterogeneous AI chip scheduling method based on unified runtime and performance prediction provided in this application first converts the user-submitted unmodified source code into a unified, hardware-independent intermediate representation, and then automatically compiles it into a native executable program for the target chip based on the scheduling results. This overcomes the compatibility barriers of the underlying software stacks between different chips and solves the technical problem in existing technologies where AI task code needs to be rewritten or adapted for different chips. Furthermore, by deeply analyzing the computation graph characteristics of the task and accurately predicting multi-dimensional execution performance indicators for each available AI chip in the cluster, it provides clear and quantitative data for scheduling decisions, solving the technical problem of insufficient decision-making basis due to inaccurate performance evaluation. Therefore, by deeply integrating the seamless execution capability of the code with accurate performance prediction decisions, all computing resources in the heterogeneous computing cluster can be included in a unified scheduling scope and optimally allocated, thereby significantly improving the overall resource scheduling efficiency.

[0161] Figure 3 This is a schematic diagram of a heterogeneous AI chip scheduling system based on unified runtime and performance prediction, provided as an embodiment of this application.

[0162] This invention provides a heterogeneous AI chip scheduling system based on unified runtime and performance prediction. This system is applied to a heterogeneous computing cluster composed of various types of AI chips, aiming to automatically and efficiently schedule user-submitted AI tasks to the most suitable chip for execution. Figure 3 As shown, the system mainly consists of three core modules: a unified runtime abstraction layer, a performance prediction model, and an intelligent scheduler.

[0163] The performance prediction model is a machine learning model trained through supervised learning, such as a model composed of a gradient boosting decision tree and a long short-term memory network. First, a training sample set containing task feature vectors, real-time chip status information, and actual performance metrics is constructed by continuously collecting historical execution data of completed tasks in the cluster. Then, the model is trained through an iterative optimization process: the training samples are input into the model to obtain prediction results; the difference between the prediction and the actual result is calculated using a loss function; and the model's internal parameters are adjusted based on this difference using optimization algorithms such as gradient descent until the model converges, ultimately resulting in a well-trained performance prediction model.

[0164] Performance prediction models are the data foundation for intelligent decision-making. Their core function is to receive the features of an AI task and the real-time state of a specific chip, and accurately predict multiple performance metrics of the task when executed on that chip. Before prediction, a standardized input feature needs to be generated for the AI ​​task. This corresponds to step S120 in the method of this application, and a detailed implementation thereof. This process generates a computational graph by tracing the source code, extracts original features such as the number of operators and tensor size, and further calculates derived features such as computational density and memory access patterns. Finally, all features are encoded and concatenated into a fixed-dimensional task feature vector.

[0165] After obtaining the task feature vector, the performance prediction model begins to execute its core functions. This corresponds to step S130 in the method of this application. The model traverses all available AI chip instances in the cluster, combines the task feature vector with the real-time status information of each chip, and predicts multiple execution indicators such as the estimated execution time and cost for each chip instance to execute the task. Finally, all prediction results are summarized into a multi-dimensional performance prediction matrix.

[0166] The intelligent scheduler is a decision-making component integrated into a cluster management system. It incorporates a multi-objective optimization and evaluation algorithm. The core function of the intelligent scheduler is to select the optimal execution chip for AI tasks based on the decision criteria provided by the performance prediction model and in conjunction with the user's business needs.

[0167] Multi-objective optimization evaluation is the core decision-making process of the intelligent scheduler, corresponding to step S140 in the method of this invention, and a specific implementation thereof. This process first receives a multi-dimensional performance prediction matrix output by the performance prediction model. Then, it introduces an additional decision dimension: a memory matching score. This score is calculated by evaluating the peak memory requirements of the task, calculating the memory access ratio, and assessing the compatibility between the memory access ratio and the physical memory capacity and bandwidth of each chip. Next, based on the user-specified execution priority, the indicators in the multi-dimensional performance prediction matrix are normalized and weighted to obtain a preliminary comprehensive evaluation score. Finally, this comprehensive evaluation score is combined with the memory matching score to obtain the final scheduling score, and the AI ​​chip with the highest score is selected as the target AI chip, generating a chip allocation decision including its physical location identifier.

[0168] The Unified Runtime Abstraction Layer (UNLI) is a software stack containing multiple backend compilers, deployed on every compute node in the cluster. Its core function is to receive decisions from the intelligent scheduler and automatically and efficiently convert user-submitted, unmodified source code into native executable programs that can run on the target AI chip, thus shielding the heterogeneity of the underlying hardware.

[0169] Code transformation and compilation are core functions of the unified runtime abstraction layer, corresponding to step S150 in the method of this application, and a specific implementation thereof. This process first receives chip allocation decisions generated by the intelligent scheduler. Then, it transforms the source code into a unified, hardware-independent intermediate representation. During the compilation of this intermediate representation, it executes an intelligent, differentiated strategy:

[0170] For the most computationally intensive performance-critical operators in the AI ​​task, an automatic tuning process is initiated. This involves searching and benchmarking to find the optimal target operator implementation on the target AI chip, and then replacing the standard function mapping by embedding this optimal implementation at compile time. For other ordinary function calls in the code, standard runtime function-to-native function mappings are performed. Finally, by linking all optimized and standard code, a high-performance native executable package tailored to the target AI chip is generated.

[0171] After generating the native executable package, the unified runtime abstraction layer is also responsible for the final deployment of the task. This corresponds to step S160 in the method of this application. Based on the node and device identifier in the chip allocation decision, it creates an isolated environment such as a container on the specified computing node, mounts the target AI chip, and loads and starts the native executable package.

[0172] Figure 4A schematic diagram of another heterogeneous AI chip scheduling system based on unified runtime and performance prediction provided in this application embodiment is shown below. Figure 4 As shown, the system may include:

[0173] The acquisition module 410 is used to acquire an AI task request including source code and real-time status information of each AI chip in the computing cluster. The AI ​​task request includes the execution priority.

[0174] The generation module 420 is used to parse the source code to generate computation graph data and extract operator type and tensor shape information from the computation graph data to generate task feature vectors.

[0175] The prediction module 430 is used to input the task feature vector and the real-time status information of each AI chip into the trained performance prediction model to obtain multiple execution indicators of each AI chip executing AI task requests, and generate a multi-dimensional performance prediction matrix based on the multiple execution indicators.

[0176] The determination module 440 is used to perform multi-objective optimization evaluation based on the multi-dimensional performance prediction matrix and execution priority to determine the target AI chip in the computing cluster and generate a chip allocation decision that includes the node and device identifiers of the target AI chip.

[0177] The generation module 420 is also used to convert the source code into a unified hardware-independent intermediate representation, map the runtime functions in the intermediate representation to the native functions of the target AI chip, and compile them into a native executable package. During the compilation process, the communication library calls of the distributed task are replaced with preset unified communication interface calls.

[0178] The deployment module 450 is used to locate the target AI chip in the computing cluster based on the node and device identifiers in the chip allocation decision, and to deploy and execute the native executable package on the target AI chip.

[0179] The heterogeneous AI chip scheduling system based on unified runtime and performance prediction in this application is used to implement the aforementioned heterogeneous AI chip scheduling method based on unified runtime and performance prediction. Therefore, the specific implementation of the heterogeneous AI chip scheduling system based on unified runtime and performance prediction can be found in the embodiment section of the heterogeneous AI chip scheduling method based on unified runtime and performance prediction above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0180] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application is shown.

[0181] The electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0182] Specifically, the processor 510 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0183] Memory 520 may include mass storage for data or instructions. For example, and not limitingly, memory 520 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 520 is non-volatile solid-state memory.

[0184] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this disclosure.

[0185] The processor 510 reads and executes computer program instructions stored in the memory 520 to implement any of the heterogeneous AI chip scheduling methods based on unified runtime and performance prediction in the above embodiments.

[0186] In one example, the electronic device may also include a communication interface 530 and a bus 540. Wherein, such as Figure 5 As shown, the processor 510, memory 520, and communication interface 530 are connected through bus 540 and complete communication with each other.

[0187] The communication interface 530 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0188] Bus 540 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 540 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0189] The electronic device can execute the heterogeneous AI chip scheduling method based on unified runtime and performance prediction in the embodiments of this application, thereby realizing the heterogeneous AI chip scheduling method based on unified runtime and performance prediction described in conjunction with the accompanying drawings.

[0190] Furthermore, in conjunction with the heterogeneous AI chip scheduling method based on unified runtime and performance prediction in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the heterogeneous AI chip scheduling methods based on unified runtime and performance prediction in the above embodiments.

[0191] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0192] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0193] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0194] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0195] The foregoing has provided a detailed description of a heterogeneous AI chip scheduling method, system, electronic device, and storage medium based on unified runtime and performance prediction provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A heterogeneous AI chip scheduling method based on unified runtime and performance prediction, applied to computing clusters including multiple types of AI chips, characterized in that, The method includes: Obtain an AI task request that includes source code and real-time status information of each AI chip in the computing cluster, wherein the AI ​​task request includes execution priority; The source code is parsed to generate computation graph data, and operator type and tensor shape information are extracted from the computation graph data to generate task feature vectors; The task feature vector and the real-time state information of each AI chip are input into the trained performance prediction model to obtain multiple execution indicators for each AI chip to execute the AI ​​task request, and a multi-dimensional performance prediction matrix is ​​generated based on the multiple execution indicators. Based on the multidimensional performance prediction matrix and the execution priority, a multi-objective optimization evaluation is performed to determine the target AI chip in the computing cluster and generate a chip allocation decision that includes the node and device identifiers of the target AI chip. After converting the source code into a unified, hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to the native functions of the target AI chip and compiled to generate a native executable package. During the compilation process, the communication library calls of the distributed task are replaced with preset unified communication interface calls. Based on the node and device identifiers in the chip allocation decision, the target AI chip is located in the computing cluster, and the native executable package is deployed and executed on the target AI chip.

2. The method according to claim 1, characterized in that, The method further includes: Obtain memory architecture data for each AI chip, including memory bandwidth and memory capacity; By determining the tensor size and operator type in the computation graph data, the peak memory usage and computation-to-memory ratio of the AI ​​task request are calculated. A memory matching score is calculated for each AI chip by evaluating the compatibility between the peak memory usage and the memory capacity, as well as the matching degree between the computational memory access ratio and the memory bandwidth. The step of performing multi-objective optimization evaluation based on the multi-dimensional performance prediction matrix and the execution priority to determine the target AI chip in the computing cluster includes: Using the memory matching score as a weighting factor, a multi-objective optimization evaluation is performed based on the multi-dimensional performance prediction matrix and the execution priority to determine the target AI chip in the computing cluster.

3. The method according to claim 2, characterized in that, The step of using the memory matching score as a weighting factor, and performing multi-objective optimization evaluation based on the multi-dimensional performance prediction matrix and the execution priority to determine the target AI chip in the computing cluster includes: The execution metrics of each AI chip in the multidimensional performance prediction matrix are normalized within the range of the corresponding execution metrics of all AI chips to generate a normalized metric matrix. The normalized indexes in the normalized index matrix are assigned evaluation weights according to the execution priority, and the normalized indexes of each AI chip are multiplied by the corresponding evaluation weights and then summed to calculate the comprehensive evaluation score of each AI chip. The execution priority is at least one of completion time priority, resource utilization priority, and energy consumption cost priority. The overall evaluation score of each AI chip is multiplied by the memory matching score to obtain a scheduling score, and the AI ​​chip with the highest scheduling score is selected as the target AI chip.

4. The method according to claim 1, characterized in that, The method further includes: By calculating the computational cost of each operator in the computation graph data, the performance-critical operators with the largest proportion of computational cost in the total computational cost are identified. For the aforementioned performance-critical operators, a variety of alternative operators adapted to the target AI chip are generated by systematically sampling within a search space consisting of tiling size and memory access patterns. Benchmark tests are performed on the various candidate operator implementations on the target AI chip to measure their respective execution times, and the candidate operator implementation with the shortest execution time is selected as the target operator implementation; The step of converting the source code into a hardware-independent intermediate representation, mapping the runtime functions in the intermediate representation to native functions of the target AI chip, and compiling to generate a native executable package includes: After converting the source code into a unified, hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to the native functions of the target AI chip. During the compilation process, the standard function mappings of the performance-critical operators are replaced by embedding the implementation of the target operators, and finally the native executable package is generated.

5. The method according to claim 4, characterized in that, After converting the source code into a unified, hardware-independent intermediate representation, the runtime functions in the intermediate representation are mapped to the native functions of the target AI chip. During compilation, the standard function mappings for the performance-critical operators are replaced by embedding the implementation of the target operators, ultimately generating the native executable package, including: An abstract syntax tree is constructed by performing syntactic and semantic analysis on the source code, and a unified, hardware-independent intermediate representation is generated based on the abstract syntax tree. Traverse all function calls in the intermediate representation and identify the target function call corresponding to the performance key operator by comparing the name of each function call with the name of the performance key operator; For the target function call, the compilation path that generates the standard native function call is replaced by directly embedding the instruction sequence of the target operator implementation; For function calls in the intermediate representation other than the target function call, a standard runtime function-to-native function mapping is performed to generate a standard native function call; The embedded instruction sequence and the standard native function calls are linked to generate the native executable package.

6. The method according to claim 1, characterized in that, The process of parsing the source code to generate computation graph data and extracting operator type and tensor shape information from the computation graph data to generate task feature vectors includes: By tracing the execution flow of the source code on the virtual device, the source code is converted into a computation graph data consisting of operator nodes representing computational operations and tensor edges representing data flows. Traverse all operator nodes and tensor edges in the computation graph data to count the number of different types of operator nodes and record the shape and size of each tensor edge; Based on the number of different types of operator nodes and the shape and size of each tensor edge, the computational density and memory access pattern of the AI ​​task request are calculated, and a structured feature set is constructed using the computational density, the memory access pattern, the number of different types of operator nodes, and the shape and size of each tensor edge. The number of different types of operator nodes, the shape and size of each tensor edge, the computational density, and the memory access pattern in the structured feature set are converted and concatenated using preset numerical encoding rules to generate the task feature vector.

7. The method according to claim 1, characterized in that, Before inputting the task feature vector and the real-time state information of each AI chip into the trained performance prediction model to obtain the multi-dimensional performance prediction matrix, the method includes: Obtain a training sample set, which includes multiple training samples. Each training sample includes a task feature vector generated by a historical AI task request, the real-time status information of the AI ​​chip at the time the historical AI task request was executed, and the corresponding real execution metrics. For each training sample in the training sample set, perform the following steps: The task feature vector and real-time status information of each training sample are input into a preset performance prediction model to obtain the predicted execution index; The loss function value of the performance prediction model is determined based on the actual performance metric and the predicted performance metric of each training sample. If the loss function value does not meet the preset training stopping condition, the model parameters of the performance prediction model are adjusted to obtain the updated performance prediction model. Then, the task feature vector and real-time state information in each training sample are input into a preset performance prediction model to obtain the predicted execution index until the loss function value meets the training stopping condition, thus obtaining the trained performance prediction model.

8. A heterogeneous AI chip scheduling system based on unified runtime and performance prediction, applied to computing clusters including multiple types of AI chips, characterized in that, The system includes: The acquisition module is used to acquire an AI task request including source code and real-time status information of each AI chip in the computing cluster. The AI ​​task request includes the execution priority. The generation module is used to parse the source code to generate computation graph data, and extract operator type and tensor shape information from the computation graph data to generate task feature vectors; The prediction module is used to input the task feature vector and the real-time status information of each AI chip into the trained performance prediction model to obtain multiple execution indicators for each AI chip to execute AI task requests, and generate a multi-dimensional performance prediction matrix based on the multiple execution indicators. The determination module is used to perform multi-objective optimization evaluation based on the multi-dimensional performance prediction matrix and the execution priority to determine the target AI chip in the computing cluster and generate a chip allocation decision including the node and device identifiers of the target AI chip. The generation module is also used to convert the source code into a unified hardware-independent intermediate representation, map the runtime functions in the intermediate representation to the native functions of the target AI chip, and compile them into a native executable package. During the compilation process, the communication library calls of the distributed task are replaced with preset unified communication interface calls. The deployment module is used to locate the target AI chip in the computing cluster based on the node and device identifier in the chip allocation decision, and to deploy and execute the native executable package on the target AI chip.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the heterogeneous AI chip scheduling method based on unified runtime and performance prediction as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the heterogeneous AI chip scheduling method based on unified runtime and performance prediction as described in any one of claims 1 to 7.

Images