Adaptive model partitioning deployment methods, systems, media, and computer products for heterogeneous hardware
By constructing hardware capability vectors and a unified intermediate representation, and dynamically segmenting and scheduling deep learning models, the problem of insufficient utilization of computing resources in heterogeneous hardware environments is solved, and efficient, low-latency cross-hardware collaborative computing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING BITMAP INFORMATION TECH CO LTD
- Filing Date
- 2026-04-02
- Publication Date
- 2026-06-30
AI Technical Summary
Existing technologies cannot dynamically adjust the computation graph structure and task allocation strategy in heterogeneous hardware environments, resulting in insufficient utilization of computing power on high-performance hardware or inability to meet real-time requirements on resource-constrained hardware, and thus failing to achieve collaborative computing and optimal resource matching across heterogeneous hardware.
By constructing hardware capability vectors, generating a unified intermediate representation, compiling it into native instruction fragments, and segmenting the computation graph into subgraphs in real time for scheduling, pipelined execution across hardware is achieved, dynamically adapting to load conditions.
It improves the computing efficiency and resource utilization of heterogeneous hardware clusters, reduces communication overhead and execution latency, and enables efficient cross-hardware collaborative inference.
Smart Images
Figure CN122309164A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of model deployment technology, and in particular to an adaptive model partitioning deployment method, system, medium, and computer product for heterogeneous hardware. Background Technology
[0002] With the increasing complexity of deep learning models and the diversification of application scenarios, efficiently deploying trained models to real-world hardware environments for computational inference has become a key challenge for both industry and academia. Especially in scenarios such as edge computing, IoT terminals, and cloud computing data centers, hardware platforms typically exhibit significant heterogeneity, containing multiple computing units with varying architectures, computing power, and power consumption characteristics, such as general-purpose CPUs, GPUs with strong parallel computing capabilities, and NPUs specializing in specific tensor computations.
[0003] To reduce the computational resource requirements of models and adapt to resource-constrained or high-efficiency scenarios, model compression and lightweighting techniques have become mainstream pre-deployment optimization solutions. These techniques typically include knowledge distillation, network pruning, and weight quantization. The core idea is to obtain a lightweight model with a more streamlined structure, fewer parameters, and lower computational load through algorithmic processing during the offline phase. This model is then deployed as a complete, immutable computational unit to the target device for execution. This approach reduces the model's computational resource requirements to some extent. However, it has significant limitations: First, the model's structure and computational load are fixed before deployment, representing a static optimization. Once the hardware type, computing power distribution, or concurrent load in the deployment environment changes, this static model cannot dynamically adjust its computational graph structure or task allocation strategy, potentially leading to insufficient computing power utilization on high-performance hardware or failure to meet real-time requirements on resource-constrained hardware. Second, this approach is usually based on the assumption of homogeneous hardware, treating the model as a whole deployed on a single type of computational unit, which fails to achieve collaborative computing and optimal resource matching across heterogeneous hardware.
[0004] Therefore, there is an urgent need to provide an adaptive model partitioning and deployment method for heterogeneous hardware, which can sense and quantify the hardware capabilities of each computing unit, and then dynamically partition and schedule the computing graph according to the load status of each computing unit in the heterogeneous hardware cluster, thereby improving the overall computing efficiency and resource utilization of the heterogeneous hardware cluster. Summary of the Invention
[0005] This invention provides an adaptive model partitioning deployment method, system, medium, and computer product for heterogeneous hardware. It can sense and quantify the hardware capabilities of each computing unit, and then dynamically partition and schedule the computing graph according to the load status of each computing unit in the heterogeneous hardware cluster, thereby improving the overall computing efficiency and resource utilization of the heterogeneous hardware cluster.
[0006] The basic solution provided by this invention is as follows:
[0007] An adaptive model partitioning deployment method for heterogeneous hardware includes the following steps:
[0008] S100: Real-time acquisition of hardware feature data of each computing unit in the heterogeneous hardware cluster, and construction of hardware capability vectors corresponding to each computing unit based on the hardware feature data.
[0009] S200, convert the deep learning model to be deployed into a unified intermediate representation; the unified intermediate representation is used to describe the structure of the computation graph corresponding to the deep learning model;
[0010] S300, based on the hardware capability vector, compile the unified intermediate representation into native instruction fragments corresponding to each computing unit, and construct a pre-compiled instruction library;
[0011] S400: Receive the real-time inference task and analyze the load status of each computing unit; based on the constraints of the real-time inference task and the load status of each computing unit, divide the computation graph corresponding to the unified intermediate representation into several subgraphs and assign a target computing unit to each subgraph.
[0012] S500: Bind each subgraph to the corresponding target computing unit and the corresponding native instruction fragment in the pre-compiled instruction library and distribute them for execution, outputting the inference results.
[0013] Furthermore, the hardware feature data includes static hardware parameters and dynamic performance data.
[0014] Furthermore, S100 includes:
[0015] S101, obtain the static hardware parameters of each computing unit;
[0016] S102, inject a preset benchmark test load into each computing unit to obtain dynamic performance data of each computing unit;
[0017] S103, quantize the static hardware parameters and dynamic performance data to generate a hardware capability vector.
[0018] Furthermore, the static hardware parameters include one or more of the following: memory capacity, peak computing performance, and supported data precision types.
[0019] Furthermore, the dynamic performance data includes one or more of actual throughput, latency, and operator execution efficiency scores.
[0020] Furthermore, the hardware capability vector is represented as follows:
[0021]
[0022] In the formula, Indicates the first Hardware capability vector of each computing unit. For the first The actual throughput of each computing unit For the first The latency of each computing unit For the first Memory capacity of each computing unit For the first Peak computational performance of each computing unit For the first The set of data precision types supported by each computing unit For the first The operator execution efficiency score for each computational unit.
[0023] Furthermore, the S200 includes:
[0024] S201, the deep learning model is parsed into a computation graph; the computation graph consists of several operators and several edges representing data flow dependencies;
[0025] S202, convert each operator in the computation graph into a preset standardized description form;
[0026] S203 generates a unified intermediate representation based on the standardized description form of each operator and the data flow dependency relationship of each operator in the computation graph.
[0027] Furthermore, the preset standardized description form is a hierarchical tensor descriptor, which is used to encapsulate the input tensor dimension, output tensor dimension, and computation logic of each operator.
[0028] Furthermore, in the unified intermediate representation, the standardized description of the operator is expressed as:
[0029]
[0030] In the formula, Operator Standardized descriptors, For operators The input tensor dimension tuple, For operators The output tensor dimension tuple, For operators The identifier for the computational logic function.
[0031] Furthermore, the S300 includes:
[0032] S301, for each operator in the unified intermediate representation, based on the hardware capability vector of each computing unit in the heterogeneous hardware cluster, the operator is compiled into native instruction fragments adapted to the architecture of different computing units. The compilation process is represented as follows:
[0033]
[0034] In the formula, Represented as the first Compiled by each computing unit, corresponding to the operator The original instruction fragment; Operator Standardized descriptors; Indicates the first Hardware capability vector of each computing unit; This is a compilation function, representing a function that combines operator computation logic with hardware capability vectors for compilation. Indicates the operator Computational logic and hardware capability vector Combine.
[0035] Furthermore, the S300 also includes:
[0036] S302, obtain the type of the computing unit corresponding to the native instruction fragment, the type of the operator, and the data precision of the native instruction fragment;
[0037] S303, using the type of the computing unit, the type of the operator, and the data precision of the native instruction fragment as a joint index, a pre-compiled instruction library is constructed to store all the native instruction fragments.
[0038] Furthermore, the S400 includes:
[0039] S401, receives real-time inference tasks;
[0040] S402, Obtain resource utilization data for each computing unit and analyze the load status of each computing unit;
[0041] S403, Based on the constraints of the real-time inference task and the load status of each computing unit, the computation graph is dynamically segmented to generate several sub-graphs;
[0042] S404: Obtain the computational characteristics of each subgraph, and determine the target computational unit corresponding to each subgraph based on the computational characteristics of each subgraph and the hardware capability vector corresponding to each computational unit.
[0043] Furthermore, the load status includes the current remaining available computing power and load level of the computing unit.
[0044] Furthermore, S402 includes: real-time monitoring and acquisition of resource utilization data of each computing unit; and calculation of the current remaining available computing power and load level of each computing unit in combination with the actual throughput and latency in the hardware capability vector.
[0045] Furthermore, S403 includes: segmenting the computation graph with the goal of minimizing the end-to-end total latency of the real-time inference task, while satisfying the constraints.
[0046] Furthermore, the constraint condition includes: the amount of data transmission between any two adjacent subgraphs after segmentation is less than a preset communication threshold.
[0047] Furthermore, the computational features include one or more of the following: the total computational cost of the operators contained in the subgraph, memory access pattern features, and internal parallelism.
[0048] Furthermore, S404 includes:
[0049] S4041, Obtain the computational features of each subgraph;
[0050] S4042, Based on the computational characteristics of the subgraph and the hardware capability vector corresponding to each computing unit, evaluate the matching degree between the subgraph and each computing unit;
[0051] S4043, the calculation unit with the highest matching degree is selected as the target calculation unit of the subgraph.
[0052] Furthermore, the analysis of the end-to-end total latency includes the execution time of each subgraph on each computing unit and the cross-unit communication time between subgraphs.
[0053] Furthermore, the S500 includes:
[0054] S501, bind each subgraph to the corresponding target computing unit in the pre-compiled instruction library to generate a schedulable task;
[0055] S502 distributes each schedulable task to the corresponding target computing unit;
[0056] S503, each target computing unit loads and executes the native instruction fragment in the schedulable task, calculates the corresponding subgraph in sequence, and transmits the intermediate data output by the subgraph to the computing unit where the downstream subgraph is located, until all subgraphs are calculated and the final inference result is output.
[0057] Further, S501 includes: retrieving and combining corresponding native instruction fragments from the pre-compiled instruction library according to the operators contained in the subgraph and the type of the target computing unit, to generate a schedulable task corresponding to the subgraph.
[0058] Furthermore, in S503, when the target computing unit loads and executes the corresponding native instruction fragment, it directly maps the data to be processed to the execution address space of the loaded native instruction fragment through a memory mapping mechanism; the data to be processed is the input data of the deep learning model for the first subgraph that executes a schedulable task, and the intermediate data output by the upstream subgraph for other subgraphs.
[0059] Furthermore, in S503, the target computing unit transmits the intermediate data output by the subgraph to the computing unit where the downstream subgraph is located through a remote direct memory access mechanism.
[0060] The second basic solution provided by this invention: an adaptive model partitioning deployment system for heterogeneous hardware, comprising:
[0061] The hardware awareness module is used to acquire hardware feature data of each computing unit in the heterogeneous hardware cluster, and construct a hardware capability vector corresponding to each computing unit based on the hardware feature data.
[0062] The model compilation module is used to convert the deep learning model to be deployed into a unified intermediate representation; the unified intermediate representation is used to describe the structure of the computation graph corresponding to the deep learning model; it is also used to compile the unified intermediate representation into native instruction fragments corresponding to each computing unit according to the hardware capability vector, and build a pre-compiled instruction library.
[0063] An adaptive scheduling module is used to receive real-time inference tasks, analyze the load status of each computing unit, and divide the computation graph corresponding to the unified intermediate representation into several subgraphs according to the constraints of the real-time inference tasks and the load status of each computing unit, and assign a target computing unit to each subgraph.
[0064] The distributed execution module is used to bind each subgraph with the corresponding target computing unit to the native instruction fragments in the pre-compiled instruction library and distribute them for execution, outputting inference results.
[0065] Furthermore, the adaptive scheduling module is also used to collect the running status data of each computing unit in the heterogeneous hardware cluster, and analyze whether there are any abnormalities in each computing unit based on the running status data, and identify abnormal computing units.
[0066] Furthermore, the operational status data includes resource utilization data and hardware health indicator data, wherein the hardware health indicator data includes one or more of the following: temperature of the computing unit, power consumption, and hardware error count.
[0067] Furthermore, the aforementioned adaptive model segmentation deployment method for heterogeneous hardware was used.
[0068] The third basic solution provided by this invention is a storage medium for storing computer-executable instructions, which, when executed, implement the above-mentioned adaptive model partitioning and deployment method for heterogeneous hardware.
[0069] The fourth basic solution provided by this invention is a computer program product, which includes a computer program that, when executed by a processor, implements the above-mentioned adaptive model partitioning and deployment method for heterogeneous hardware.
[0070] The principles and advantages of this invention are as follows:
[0071] 1. This solution achieves precise quantification and dynamic perception of computing resources in heterogeneous hardware clusters by constructing hardware capability vectors. Specifically, the system acquires hardware characteristic data of each computing unit and constructs hardware capability vectors accordingly. This quantifies key attributes such as actual throughput, latency, memory capacity, and peak computing performance of each computing unit, providing a dynamic and quantitative decision-making basis for subsequent model segmentation and subgraph scheduling. This overcomes the shortcomings of static deployment solutions in being insensitive to environmental changes and is conducive to optimizing the utilization of heterogeneous hardware cluster resources.
[0072] 2. By generating a unified intermediate representation and building a pre-compiled instruction library, hardware-independent representation and instruction-level adaptation for model computation are achieved. This solution first parses the target model into a computation graph and converts it into a hardware-independent unified intermediate representation, abstracting the differences between different frameworks. Then, based on the hardware capability vector, each operator in the unified intermediate representation is compiled into native instruction fragments for different computing unit architectures, and a pre-compiled instruction library is built, indexed by hardware type, operator type, and data precision. This mechanism ensures that optimized native instruction fragments can be directly loaded during inference execution, avoiding runtime format conversion and recompilation overhead, and improving execution efficiency for cross-platform deployment.
[0073] 3. Based on real-time load status-based dynamic partitioning and scheduling decisions, intelligent splitting and load balancing of the computation graph are achieved. After receiving real-time inference tasks, the system analyzes the load status of each computing unit and, based on task constraints, dynamically searches for the optimal partitioning point of the computation graph, dividing it into several subgraphs with the goal of minimizing the total end-to-end latency. Simultaneously, based on the computational characteristics and hardware capability vectors of the subgraphs, the system assigns a most suitable target computing unit to each subgraph. This achieves dynamic optimization of task allocation and automatic load balancing, effectively improving system response speed and resource utilization.
[0074] 4. Efficient and low-latency inference across hardware subgraphs is achieved through instruction binding and pipelined execution mechanisms. Each segmented subgraph is bound to its corresponding native instruction fragment in the pre-compiled instruction library, generating schedulable tasks and distributing them. The target computing unit directly loads and executes instructions via memory mapping and efficiently transfers intermediate data between units through remote direct memory access, forming a pipelined execution chain across hardware. This reduces communication overhead and execution latency between subgraphs, ensuring high overall performance for collaborative inference across heterogeneous hardware clusters.
[0075] In summary, this solution achieves dynamic adaptive deployment of deep learning models on heterogeneous hardware clusters through the synergy of a complete technology chain, including hardware capability vector construction, unified intermediate representation conversion, pre-compiled instruction library generation, dynamic segmentation and scheduling based on load status, and pipelined execution. This improves the overall computing efficiency and resource utilization of heterogeneous hardware clusters. Attached Figure Description
[0076] Figure 1 This is a flowchart of the adaptive model segmentation and deployment method for heterogeneous hardware according to the present invention. Detailed Implementation
[0077] The following detailed description illustrates the specific implementation method:
[0078] Example 1:
[0079] Adaptive model partitioning deployment methods for heterogeneous hardware, such as Figure 1 As shown, this method achieves efficient and adaptive deployment of deep learning models on heterogeneous computing unit clusters including CPUs, GPUs, and NPUs through the following steps:
[0080] S100: Real-time acquisition of hardware characteristic data of each computing unit in the heterogeneous hardware cluster, and construction of a hardware capability vector corresponding to each computing unit based on the hardware characteristic data. The hardware characteristic data includes static hardware parameters and dynamic performance data. S100 includes:
[0081] S101, Obtain the static hardware parameters of each computing unit. These static hardware parameters include one or more of the following: memory capacity, peak computing performance, and supported data precision types. Specifically, during deployment initialization, the system automatically detects each computing unit (hereinafter referred to as an index) by calling standard interfaces provided by the operating system and hardware manufacturers. This refers to the static hardware parameters (represented by the CPU). In this embodiment, the static hardware parameters include memory capacity, i.e., the total available memory of the device, in bytes; peak computing performance, i.e., the theoretical maximum computing power of the device, specifically the number of floating-point operations per second (Floating-Point Operations Per Second) of the CPU, obtained from the hardware specification table, in the number of floating-point operations per second; and the set of supported data precision types, such as FP32 (single precision), FP16 (half precision), and INT8 (8-bit integer), obtained by querying through the driver or compilation toolchain. In this embodiment, the static hardware parameters also include: hardware type, such as general-purpose CPU, graphics processing unit (GPU), and neural network processor (NPU), obtained by parsing device driver information; and instruction set architecture, such as the AVX2 / AVX-512 vector instruction set supported by the x86-64 architecture, the NEON / SVE instruction set supported by the ARM architecture, and the CUDA Compute Capability version supported by the NVIDIA GPU, obtained through the CPUID instruction, system files (such as / proc / cpuinfo), or GPU driver APIs (such as cudaGetDeviceProperties).
[0082] S102, inject a preset benchmark load into each computing unit to obtain dynamic performance data for each computing unit. The dynamic performance data includes one or more of actual throughput, latency, and operator execution efficiency scores. Specifically, to obtain the hardware's performance under actual load, the system injects a set of preset benchmark loads into each computing unit. The benchmark load consists of micro-computing kernels covering typical deep learning operations, including convolutions, matrix multiplications, pooling, and activation functions of different sizes. Through multiple executions and timing, the following dynamic performance data is measured: actual throughput, i.e., the number of standard computational operations (Ops / s) that the hardware can complete per unit time, which is the actual computing power after considering factors such as memory bandwidth and caching effectiveness; latency; operator execution efficiency score, which is a vector, where the... Each component This indicates that the hardware is for the first The execution efficiency of operators such as convolution can be calculated in one specific way: That is, using this hardware to The actual throughput of a class of operators is normalized by dividing the maximum throughput of all tested hardware against the same class of operators. This score quantifies the relative advantage of the hardware against a specific operator.
[0083] S103, the static hardware parameters and dynamic performance data are quantified to generate a hardware capability vector, which is a quantified profile of the comprehensive capabilities of the computing unit. The hardware capability vector is represented as:
[0084]
[0085] In the formula, Indicates the first Hardware capability vector of each computing unit. For the first The actual throughput of each computing unit For the first The latency of each computing unit For the first Memory capacity of each computing unit For the first Peak computational performance of each computing unit For the first The set of data precision types supported by each computing unit For the first The operator execution efficiency score for each computational unit.
[0086] S200, converting the deep learning model to be deployed into a unified intermediate representation; the unified intermediate representation is used to describe the structure of the computation graph corresponding to the deep learning model; S200 includes:
[0087] S201, the deep learning model is parsed into a computation graph; the computation graph consists of several operators and several edges representing data flow dependencies. Specifically, regardless of whether the original model format is TensorFlow's SavedModel, PyTorch's TorchScript, or ONNX, the system uses the corresponding model parser to convert it into a directed computation graph, where nodes represent operators and edges represent the flow direction and dependencies of tensor data.
[0088] S202, the operators in the computation graph are transformed into a preset standardized description form; the preset standardized description form is a hierarchical tensor descriptor, used to encapsulate the input tensor dimension, output tensor dimension, and computational logic of each operator. In the unified intermediate representation, the standardized description form of the operator is represented as:
[0089]
[0090] In the formula, Operator Standardized descriptors, For operators The input tensor dimension tuple, For operators The output tensor dimension tuple, For operators The identifiers for computational logic functions, such as "Conv2D" and "MatMul".
[0091] S203 generates a unified intermediate representation based on the standardized description forms of each operator and the data flow dependencies of each operator in the computation graph. That is, it is based on the standardized descriptors of all operators. Furthermore, strictly adhering to the topological connections of the original computation graph, a hardware-independent unified intermediate representation (UIR) is reconstructed and generated. The UIR is an intermediate data structure independent of any specific hardware and front-end framework, fully preserving the model's computational logic and data flow.
[0092] Furthermore, to accelerate runtime decision-making, the system constructs an "operator type-hardware type" performance lookup table during offline phases or initialization, utilizing the operator execution efficiency scores obtained in S102. This table records the operator execution efficiency scores for each type of operator on each type of hardware. When it is necessary to evaluate a specific operator (belonging to type ... In hardware When determining the expected execution efficiency, a quantitative reference value for adaptability can be obtained by directly looking up a table. .
[0093] S300, based on the hardware capability vector, compiles the unified intermediate representation into native instruction fragments corresponding to each computing unit, constructing a pre-compiled instruction library; this step utilizes a modern deep learning compiler to pre-compile the operators in the UIR into native code optimized for each hardware, constructing a fast-searchable instruction library. S300 includes:
[0094] S301, for each operator in the unified intermediate representation, based on the hardware capability vector of each computing unit in the heterogeneous hardware cluster, the operator is compiled into native instruction fragments adapted to the architecture of different computing units. The compilation process is represented as follows:
[0095]
[0096] In the formula, Represented as the first Compiled by each computing unit, corresponding to the operator The original instruction fragment; Operator Standardized descriptors; Indicates the first Hardware capability vector of each computing unit; This is a compilation function, representing a function that combines operator computation logic with hardware capability vectors for compilation. Indicates the operator Computational logic and hardware capability vector Combine.
[0097] Specifically, for each operator in UIR, the system targets each hardware type. This calls the deep learning compiler backend, such as TVM or MLIR. (Compilation function) The specific execution process is as follows: Extract The compiler extracts hardware type and instruction set architecture information, such as "NVIDIA GPU with SM 8.6" and "x86 CPU with AVX512," to determine the code generation target; extracts data precision requirements to determine the data type of computation; and, in conjunction with operator semantics, automatically performs hardware-specific advanced optimizations, such as operator fusion, loop tiling, vectorization, and memory access optimization; ultimately outputting a highly optimized native instruction fragment. Such as the CUDA kernel.
[0098] S302, obtain the type of the computing unit corresponding to the native instruction fragment, the type of the operator, and the data precision of the native instruction fragment.
[0099] S303, using the type of the computing unit, the type of the operator, and the data precision of the native instruction fragment as a joint index, a pre-compiled instruction library is constructed to store all the native instruction fragments. All native instruction fragments It is stored in a pre-compiled instruction library in versioned binary form. This design supports fast key-value retrieval at runtime and allows for dynamic updates or loading of instruction fragments with different optimizations.
[0100] S400: Receives a real-time inference task and analyzes the load status of each computing unit; based on the constraints of the real-time inference task and the load status of each computing unit, divides the computation graph corresponding to the unified intermediate representation into several subgraphs and assigns a target computing unit to each subgraph; S400 includes:
[0101] S401, Receive real-time inference task; specifically, the system receives a real-time inference request, along with the input data batch size and latency constraints.
[0102] S402, acquire resource utilization data for each computing unit and analyze the load status of each computing unit; the load status includes the current remaining available computing power and load level of the computing unit. Specifically, monitor and acquire resource utilization data for each computing unit in real time; combine the actual throughput and latency in the hardware capability vector to calculate the current remaining available computing power and load level of each computing unit. In other embodiments of this application, peak performance can also be calculated. Perform a load status assessment.
[0103] Specifically, the scheduler sends a lightweight, fixed-valued benchmark probe operator to each computing unit once per second. In this embodiment, the benchmark probe operator is a matrix multiplication with a preset size. The current execution latency of this operator is measured. Simultaneously, the latency measured during the benchmarking phase is obtained from the hardware capability vector of the computing unit. and actual throughput Current load level of the computing unit This is reflected comprehensively by the degree of degradation of the current execution latency compared to the optimal latency:
[0104]
[0105] This represents the current load level of the computing unit, indicating how much the current execution latency has increased compared to the optimal latency. This indicates that it is in its optimal performance state. The larger the value, the heavier the load and the more pronounced the performance degradation. To avoid momentary jitter, you can... Perform a moving average process.
[0106] Current remaining available computing power of the computing unit Estimate using its baseline throughput and current load level:
[0107]
[0108] S403, based on the constraints of the real-time inference task and the load status of each computing unit, the computation graph is dynamically segmented to generate several subgraphs. Specifically, with minimizing the end-to-end total latency of the real-time inference task as the optimization objective, the computation graph is segmented while satisfying the constraints. The constraints include: the data transmission volume between any two adjacent subgraphs after segmentation is lower than a preset communication threshold, and all operators within each subgraph can be completely covered by the pre-compiled instruction library of the same hardware unit. The analysis of the end-to-end total latency includes the execution time of each subgraph on each computing unit and the cross-unit communication time between subgraphs.
[0109] Specifically, construct the optimization objective function. :
[0110]
[0111] In the formula, The total end-to-end delay from input to output consists of the computation time of each subgraph and the cross-device communication time between subgraphs. Preset delay constraints.
[0112] For any dividing point The amount of data transferred between any two adjacent subgraphs after partitioning. Below the preset communication threshold :
[0113]
[0114] This constraint ensures that the partitioning does not negate the benefits of parallel computing due to excessive communication overhead.
[0115] Hardware consistency constraints: each subgraph All operators within All of them can be allocated to the hardware units The corresponding pre-compiled instruction fragment was found above, and its formal description is as follows:
[0116]
[0117] In the formula, This indicates a pre-compiled instruction library.
[0118] The system employs a dynamic programming algorithm to search for the optimal split point on the computational graph while satisfying the aforementioned constraints, thereby minimizing the total end-to-end delay. The dynamic programming algorithm treats the computational graph as a directed acyclic graph and uses state transition equations to recursively solve for the optimal splitting scheme that satisfies the constraints.
[0119] S404: Obtain the computational characteristics of each subgraph, and determine the target computational unit corresponding to each subgraph based on the computational characteristics of each subgraph and the hardware capability vector corresponding to each computational unit. Specifically, under the premise of satisfying all constraints in step S403, the scheduler allocates target computational units to each subgraph with the goal of minimizing the overall system load. This goal is formally expressed as minimizing the sum of the load levels of all computational units:
[0120]
[0121] In the formula, Let be the objective function that minimizes the sum of the load levels of all computing units. For the first The current load level of each computing unit.
[0122] To achieve this goal, the system performs the following operations:
[0123] S4041, Obtain the computational characteristics of each subgraph; the computational characteristics include one or more of the following: total computational cost of operators contained in the subgraph, memory access pattern characteristics, and internal parallelism. In this embodiment, the computational characteristics include total computational cost, which is the total number of floating-point operations required for all operators in the subgraph to complete one forward inference, obtained by summing the computational cost of each operator; memory access pattern characteristics, which are determined by analyzing the input / output tensor size, memory access rules, and operator attributes of each operator in the subgraph to determine whether the subgraph as a whole is computationally intensive (high computation / memory access ratio) or memory access intensive (low computation / memory access ratio); internal parallelism, which is evaluated by analyzing the data dependencies between operators within the subgraph to assess the degree to which the subgraph can be executed in parallel within a single computation unit. In this embodiment, it is the number of parallelizable branches.
[0124] S4042, Based on the computational features of the subgraph and the hardware capability vectors corresponding to each computational unit, evaluate the matching degree between the subgraph and each computational unit. In this embodiment, a pre-trained matching degree evaluation neural network model is used to evaluate the matching degree between the subgraph and the computational unit. The matching degree evaluation neural network model is a three-layer BP neural network model, including an input layer, a hidden layer, and an output layer. In this embodiment, the total computational cost, memory access pattern features, internal parallelism, memory capacity, peak computational performance, supported data precision type set, actual throughput, latency, and operator execution efficiency score of the operators contained in the subgraph are used as the input to the input layer. Therefore, the input layer has 9 nodes, and the output is a prediction of the matching degree score, so the output layer has 1 node. For the hidden layer, this embodiment uses the following formula to determine the number of hidden layer nodes: Where L is the number of nodes in the hidden layer, n is the number of nodes in the input layer, m is the number of nodes in the output layer, and a is a number between 1 and 10, which is taken as 6 in this embodiment. Therefore, there are 10 nodes in the hidden layer. Backpropagation (BP) neural networks typically use differentiable functions as the network's activation functions. In this embodiment, the sigmoid tangent function (tansig) is chosen as the activation function for the hidden layer neurons. The output layer neurons use the logarithmic sigmoid function (logsig) as the activation function to limit the output value to the range of 0 to 1, representing the normalized matching score.
[0125] After the BP network model is built, it is trained using historical deployment data accumulated during system operation. The label values in the training samples are represented by the ratio of actual throughput to peak throughput measured on the corresponding computing unit in the historical task. The prediction model obtained after training with historical data can achieve accurate matching degree evaluation results.
[0126] S4043, the computational unit with the highest matching degree is selected as the target computational unit of the subgraph. Specifically, among the candidate hardware set that satisfies the hardware consistency constraints in S403, the computational unit with the highest matching degree is selected for the subgraph and determined as the target computational unit.
[0127] S500 binds each subgraph to the corresponding native instruction fragment in the pre-compiled instruction library for execution and distributes them for execution, outputting the inference results. S500 includes:
[0128] S501, each subgraph is bound to the corresponding native instruction fragment in the pre-compiled instruction library with the corresponding target computing unit to generate a schedulable task. Specifically, based on the operators contained in the subgraph and the type of the target computing unit, the corresponding native instruction fragments are retrieved and combined from the pre-compiled instruction library to generate a schedulable task corresponding to the subgraph. During actual execution, for each subgraph and its target computing unit, the scheduler retrieves all corresponding native instruction fragments from the pre-compiled instruction library using (hardware type, operator type, data precision) as a joint index, based on the operator sequence contained in the subgraph. These native instruction fragments are then packaged with the input / output data descriptors (memory layout, size) of the subgraph to generate a schedulable task package.
[0129] S502, each schedulable task is distributed to its corresponding target computing unit. The scheduler distributes each schedulable task to its target computing unit. The distribution process includes transmitting the metadata of the schedulable task package and ensuring that the target computing unit can access the pre-compiled instruction fragments (stored in local or shared storage) referenced in the schedulable task package.
[0130] S503, each target computing unit loads and executes the native instruction fragments in the schedulable task, sequentially calculates the corresponding subgraphs, and transmits the intermediate data output by the subgraphs to the computing units of the downstream subgraphs, until all subgraphs are calculated, and outputs the final inference result. When the target computing unit loads and executes the corresponding native instruction fragments, it uses a memory mapping mechanism to directly map the data to be processed to the execution address space of the loaded native instruction fragments; the data to be processed is the input data of the deep learning model for the first subgraph executing the schedulable task, and the intermediate data output by the upstream subgraphs for other subgraphs. The target computing unit transmits the intermediate data output by the subgraphs to the computing units of the downstream subgraphs through a remote direct memory access mechanism.
[0131] Specifically, after receiving a schedulable task, the target computing unit performs the following operations in sequence:
[0132] The pre-compiled native instruction fragments within the task package are loaded directly from storage. A memory mapping mechanism directly maps the data to be processed into the instruction's execution address space. Since the data layout is determined at compile time (S300) and is consistent with the expected instruction format, this process skips any runtime data format conversion or recompilation.
[0133] The target computation unit executes the loaded native instruction fragments to complete the computation of the subgraph.
[0134] After a subgraph is computed, its output data is transmitted directly and asynchronously to the designated input buffer of the computation unit containing its downstream subgraph via a remote direct memory access network. This process does not require CPU involvement in data transfer, forming a pipelined execution chain across hardware.
[0135] The final inference result is output; the output of the subgraph at the end of the pipeline is the final inference result.
[0136] During operation, the system collects and monitors the operational status data of each computing unit in real time, including temperature, power consumption, and hardware error counts. By determining whether these indicators exceed corresponding preset thresholds, the system detects whether the hardware is in a faulty or abnormal load state. If a computing unit is determined to be faulty or overloaded, the system triggers an alarm and initiates fault recovery or load switching contingency plans. The central scheduler continuously receives and updates the hardware capability vectors and real-time load status of all computing units, providing a basis for dynamic scheduling decisions and allocating schedulable tasks from faulty or overloaded computing units to other computing units.
[0137] It should be noted that the steps described above in this embodiment are labeled with S100, S200, etc., solely for the purpose of clearly distinguishing the descriptions of each step, and not as an absolute limitation on the execution order of the steps. Provided that there is no logical contradiction, the execution order of some steps can be adjusted or they can be executed in parallel.
[0138] Example 2:
[0139] An adaptive model partitioning deployment system for heterogeneous hardware utilizes the aforementioned adaptive model partitioning deployment method for heterogeneous hardware. The system includes:
[0140] The hardware awareness module is used to acquire hardware feature data of each computing unit in the heterogeneous hardware cluster, and construct a hardware capability vector corresponding to each computing unit based on the hardware feature data.
[0141] The model compilation module is used to convert the deep learning model to be deployed into a unified intermediate representation; the unified intermediate representation is used to describe the structure of the computation graph corresponding to the deep learning model; it is also used to compile the unified intermediate representation into native instruction fragments corresponding to each computing unit according to the hardware capability vector, and build a pre-compiled instruction library.
[0142] An adaptive scheduling module is used to receive real-time inference tasks, analyze the load status of each computing unit, divide the computation graph corresponding to the unified intermediate representation into several subgraphs according to the constraints of the real-time inference tasks and the load status of each computing unit, and assign a target computing unit to each subgraph.
[0143] The distributed execution module is used to bind each subgraph with the corresponding target computing unit to the native instruction fragments in the pre-compiled instruction library and distribute them for execution, outputting inference results.
[0144] Example 3:
[0145] The basic principle of Example 3 is the same as that of Example 2. The difference is that the adaptive scheduling module in Example 3 is also used to collect the operating status data of each computing unit in the heterogeneous hardware cluster, and analyze whether there are any abnormalities in each computing unit based on the operating status data, and identify abnormal computing units. The operating status data includes resource utilization data and hardware health indicator data. The hardware health indicator data includes one or more of the following: temperature, power consumption, and hardware error count of the computing unit. In this example, the operating status data includes resource utilization data and hardware health indicator data. The hardware health indicator data includes the temperature, power consumption, and hardware error count of the computing unit.
[0146] Example 4:
[0147] This application provides a storage medium for storing computer-executable instructions, which, when executed, implement the aforementioned adaptive model partitioning deployment method for heterogeneous hardware. That is, if the methods described above are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0148] Example 5:
[0149] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the adaptive model partitioning deployment method for heterogeneous hardware as described in the above embodiments. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the methods provided in the above embodiments, and will not be elaborated upon here.
[0150] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.
Claims
1. An adaptive model partitioning and deployment method for heterogeneous hardware, characterized in that: Includes the following steps: S100: Real-time acquisition of hardware feature data of each computing unit in the heterogeneous hardware cluster, and construction of hardware capability vectors corresponding to each computing unit based on the hardware feature data. S200, convert the deep learning model to be deployed into a unified intermediate representation; the unified intermediate representation is used to describe the structure of the computation graph corresponding to the deep learning model; S300, based on the hardware capability vector, compile the unified intermediate representation into native instruction fragments corresponding to each computing unit, and construct a pre-compiled instruction library; S400: Receive the real-time inference task and analyze the load status of each computing unit; based on the constraints of the real-time inference task and the load status of each computing unit, divide the computation graph corresponding to the unified intermediate representation into several subgraphs and assign a target computing unit to each subgraph. S500: Bind each subgraph to the corresponding target computing unit and the corresponding native instruction fragment in the pre-compiled instruction library and distribute them for execution, outputting the inference results.
2. The adaptive model partitioning and deployment method for heterogeneous hardware according to claim 1, wherein: The hardware characteristic data includes static hardware parameters and dynamic performance data.
3. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 2, wherein: S100 includes: S101, obtain the static hardware parameters of each computing unit; S102, inject a preset benchmark test load into each computing unit to obtain dynamic performance data of each computing unit; S103, quantize the static hardware parameters and dynamic performance data to generate a hardware capability vector.
4. The adaptive model partitioning and deployment method for heterogeneous hardware of claim 2, wherein: The static hardware parameters include one or more of the following: memory capacity, peak computing performance, and the set of supported data precision types.
5. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 2, wherein: The dynamic performance data includes one or more of the following: actual throughput, latency, and operator execution efficiency score.
6. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 3, wherein: The hardware capability vector is represented as follows: In the formula, represents the hardware capability vector of the th computing unit, is the actual throughput of the th computing unit, is the delay of the th computing unit, is the memory capacity of the th computing unit, is the computing peak performance of the th computing unit, is the set of data precision types supported by the th computing unit, is the operator execution efficiency score of the th computing unit.
7. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 1, characterized in that: S200 includes: S201, the deep learning model is parsed into a computation graph; the computation graph consists of several operators and several edges representing data flow dependencies; S202, convert each operator in the computation graph into a preset standardized description form; S203 generates a unified intermediate representation based on the standardized description form of each operator and the data flow dependency relationship of each operator in the computation graph.
8. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 7, characterized in that: The preset standardized description form is a hierarchical tensor descriptor, which is used to encapsulate the input tensor dimension, output tensor dimension, and computation logic of each operator.
9. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 8, characterized in that: In the unified intermediate representation, the standardized description of the operator is expressed as: In the formula, Operator Standardized descriptors, For operators The input tensor dimension tuple, For operators The output tensor dimension tuple, For operators The identifier for the computational logic function.
10. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 1, characterized in that: The S300 includes: S301, for each operator in the unified intermediate representation, based on the hardware capability vector of each computing unit in the heterogeneous hardware cluster, the operator is compiled into native instruction fragments adapted to the architecture of different computing units. The compilation process is represented as follows: In the formula, Represented as the first Compiled by each computing unit, corresponding to the operator The original instruction fragment; Operator Standardized descriptors; Indicates the first Hardware capability vector of each computing unit; This is a compilation function, representing a function that combines operator computation logic with hardware capability vectors for compilation. Indicates the operator Computational logic and hardware capability vector Combine.
11. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 10, characterized in that: The S300 also includes: S302, obtain the type of the computing unit corresponding to the native instruction fragment, the type of the operator, and the data precision of the native instruction fragment; S303, using the type of the computing unit, the type of the operator, and the data precision of the native instruction fragment as a joint index, a pre-compiled instruction library is constructed to store all the native instruction fragments.
12. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 6, characterized in that: The S400 includes: S401 receives real-time inference tasks; S402, Obtain resource utilization data for each computing unit and analyze the load status of each computing unit; S403, Based on the constraints of the real-time inference task and the load status of each computing unit, the computation graph is dynamically segmented to generate several sub-graphs; S404: Obtain the computational characteristics of each subgraph, and determine the target computational unit corresponding to each subgraph based on the computational characteristics of each subgraph and the hardware capability vector corresponding to each computational unit.
13. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 12, characterized in that: The load status includes the current remaining available computing power and load level of the computing unit.
14. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 13, characterized in that: S402 includes: real-time monitoring and acquisition of resource utilization data for each computing unit; and calculation of the current remaining available computing power and load level of each computing unit by combining the actual throughput and latency in the hardware capability vector.
15. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 12, characterized in that: S403 includes: segmenting the computation graph with the goal of minimizing the end-to-end total latency of the real-time inference task, while satisfying the constraints.
16. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 12, characterized in that: The constraints include: the amount of data transmission between any two adjacent subgraphs after segmentation is less than a preset communication threshold.
17. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 12, characterized in that: The computational features include one or more of the following: the total computational cost of the operators contained in the subgraph, memory access pattern features, and internal parallelism.
18. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 12, characterized in that: S404 includes: S4041, Obtain the computational features of each subgraph; S4042, Based on the computational characteristics of the subgraph and the hardware capability vector corresponding to each computing unit, evaluate the matching degree between the subgraph and each computing unit; S4043, the calculation unit with the highest matching degree is selected as the target calculation unit of the subgraph.
19. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 15, characterized in that: The analysis of the total end-to-end latency includes the execution time of each subgraph on each computing unit and the cross-unit communication time between subgraphs.
20. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 11, characterized in that: The S500 includes: S501, bind each subgraph to the corresponding target computing unit in the pre-compiled instruction library to generate a schedulable task; S502 distributes each schedulable task to the corresponding target computing unit; S503, each target computing unit loads and executes the native instruction fragment in the schedulable task, calculates the corresponding subgraph in sequence, and transmits the intermediate data output by the subgraph to the computing unit where the downstream subgraph is located, until all subgraphs are calculated and the final inference result is output.
21. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 20, characterized in that: S501 includes: retrieving and combining corresponding native instruction fragments from the pre-compiled instruction library according to the operators contained in the subgraph and the type of the target computing unit, to generate a schedulable task corresponding to the subgraph.
22. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 20, characterized in that: In S503, when the target computing unit loads and executes the corresponding native instruction fragment, it directly maps the data to be processed to the execution address space of the loaded native instruction fragment through a memory mapping mechanism; the data to be processed is the input data of the deep learning model for the first subgraph that executes a schedulable task, and the intermediate data output by the upstream subgraph for other subgraphs.
23. The adaptive model partitioning deployment method for heterogeneous hardware according to claim 20, characterized in that: In S503, the target computing unit transmits the intermediate data output by the subgraph to the computing unit where the downstream subgraph is located through a remote direct memory access mechanism.
24. An adaptive model partitioning deployment system for heterogeneous hardware, characterized in that: include: The hardware awareness module is used to acquire hardware feature data of each computing unit in the heterogeneous hardware cluster, and construct a hardware capability vector corresponding to each computing unit based on the hardware feature data. The model compilation module is used to convert the deep learning model to be deployed into a unified intermediate representation; the unified intermediate representation is used to describe the structure of the computation graph corresponding to the deep learning model; it is also used to compile the unified intermediate representation into native instruction fragments corresponding to each computing unit according to the hardware capability vector, and build a pre-compiled instruction library. An adaptive scheduling module is used to receive real-time inference tasks, analyze the load status of each computing unit, divide the computation graph corresponding to the unified intermediate representation into several subgraphs according to the constraints of the real-time inference tasks and the load status of each computing unit, and assign a target computing unit to each subgraph. The distributed execution module is used to bind each subgraph with the corresponding target computing unit to the native instruction fragments in the pre-compiled instruction library and distribute them for execution, outputting inference results.
25. The adaptive model partitioning deployment system for heterogeneous hardware according to claim 24, characterized in that: The adaptive scheduling module is also used to collect the running status data of each computing unit in the heterogeneous hardware cluster, and analyze whether there are any abnormalities in each computing unit based on the running status data, and identify abnormal computing units.
26. The adaptive model partitioning deployment system for heterogeneous hardware according to claim 25, characterized in that: The operational status data includes resource utilization data and hardware health indicator data, and the hardware health indicator data includes one or more of the following: temperature of the computing unit, power consumption, and hardware error count.
27. The adaptive model partitioning deployment system for heterogeneous hardware according to claim 24, characterized in that: The adaptive model partitioning deployment method for heterogeneous hardware as described in any one of claims 1 to 23 above is used.
28. A storage medium for storing computer-executable instructions, characterized in that: When the computer-executable instructions are executed, they implement the adaptive model partitioning deployment method for heterogeneous hardware as described in any one of claims 1 to 23.
29. A computer program product, characterized in that: The computer program product includes a computer program that, when executed by a processor, implements the adaptive model partitioning deployment method for heterogeneous hardware as described in any one of claims 1 to 23.