Distributed reasoning acceleration method and system based on heterogeneous computing power dynamic allocation

Abstract

The invention discloses a distributed reasoning acceleration method and system based on heterogeneous computing power dynamic allocation, and relates to the field of computing power allocation, and the method comprises the steps: receiving a to-be-reasoned task, splitting the to-be-reasoned task into a plurality of operator subtasks, and constructing a calculation dependency graph; constructing a computing cluster containing heterogeneous devices, collecting three-dimensional data indexes of the devices in real time, and generating a dynamic resource map; constructing a task equipment matching cost matrix; carrying out optimization solution on the cost matrix to obtain an operator-level equipment distribution scheme and a cross-equipment data routing path; the to-be-reasoned task is sliced and distributed to reasoning equipment according to the operator granularity, and the actual reasoning rate of the equipment is continuously monitored in the subtask execution process; and when it is detected that the real-time reasoning rate of the target equipment is lower than a preset threshold value, triggering a dynamic redistribution mechanism. The method has the advantages that heterogeneous computing resources are monitored in real time, tasks are dynamically allocated, efficient execution of reasoning tasks is ensured, and the overall reasoning acceleration performance is improved.

Description

Technical Field

[0001] This invention relates to the field of computing power allocation, and in particular to a distributed inference acceleration method and system based on dynamic allocation of heterogeneous computing power. Background Technology

[0002] With the widespread adoption of artificial intelligence applications, deep learning models are becoming increasingly complex, and the computational load of inference tasks is increasing dramatically, making it difficult for single devices to meet the requirements for real-time performance and efficiency. Distributed inference, by allocating inference tasks across multiple computing nodes, fully utilizes cluster computing resources, significantly improving inference efficiency and processing speed. This method has been widely applied in various fields, such as video surveillance and facial recognition in intelligent security, real-time perception and decision-making in autonomous driving, and large-scale data processing in cloud computing and edge computing.

[0003] Most current distributed inference acceleration methods lack dynamic adaptability in device selection and task scheduling, often relying on static resource allocation strategies. This leads to an inability to fully utilize the computing power of heterogeneous devices when handling complex inference tasks. For example, some systems fail to monitor key information such as device computational load and memory usage in real time, resulting in device overload or resource idleness, thus affecting overall inference performance. Secondly, existing systems often neglect cross-device data transmission latency issues. When multiple devices collaborate on inference, data transmission bottlenecks are not effectively optimized, resulting in wasted network bandwidth and reduced inference efficiency. Furthermore, many methods lack dynamic adjustment mechanisms. If the inference rate of some devices falls below expectations, rapid task migration or reallocation is impossible, leading to unbalanced system load and affecting the final inference speed and accuracy. Summary of the Invention

[0004] To improve existing methods and systems, a distributed inference acceleration method and system based on dynamic allocation of heterogeneous computing power is provided. This method monitors heterogeneous computing resources in real time, dynamically allocates tasks and optimizes device usage to ensure efficient execution of inference tasks, and automatically adjusts task allocation when device load is uneven, thereby improving overall inference acceleration performance.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] Distributed inference acceleration methods based on dynamic allocation of heterogeneous computing power include:

[0007] Receive the task to be inferred, break it down into multiple operator subtasks and construct a computation dependency graph;

[0008] Construct a computing cluster containing heterogeneous devices, collect data on the computing power type, available video memory, network bandwidth, real-time computing load, and unit energy consumption of each device in real time, and generate a dynamic resource map;

[0009] Based on dynamic resource graphs and computational dependency graphs, a task-device matching cost matrix is ​​constructed.

[0010] The cost matrix is ​​optimized and solved using a greedy algorithm to obtain the operator-level device allocation scheme and cross-device data routing path;

[0011] Based on the allocation scheme, the tasks to be inferred are distributed to the inference device in slices at the operator granularity, and the actual inference rate of the device is continuously monitored during the execution of the sub-tasks.

[0012] Based on the actual inference rate of the device, when the real-time inference rate of the target device is detected to be lower than a preset threshold, a dynamic reallocation mechanism is triggered.

[0013] Preferably, the step of receiving the task to be inferred, splitting it into multiple operator subtasks, and constructing a computational dependency graph specifically includes:

[0014] Based on the computational requirements of the task to be reasoned, the task is divided into multiple independent operators, each of which is a subtask;

[0015] Based on the data flow between operators, the dependencies between operators are obtained;

[0016] A computational dependency graph is constructed based on the obtained dependencies, with operators acting as nodes in the graph and edges representing the data dependencies between operators.

[0017] Preferably, the construction of a computing cluster comprising heterogeneous devices, and the real-time collection of computing power type, available video memory, network bandwidth, real-time computing load, and unit energy consumption indicators of each device to generate a dynamic resource map specifically includes:

[0018] Construct a computing cluster containing heterogeneous devices, including CPUs, GPUs, FPGAs, TPUs, and NPUs;

[0019] Obtain computing cluster performance metrics, including computing power type, available video memory, network bandwidth, real-time computing load, and unit energy consumption metrics;

[0020] Based on the acquired data, a resource map is designed, where each node represents a device. The size, color, and shape of the node indicate the device's load, available memory, and network bandwidth information.

[0021] The graph data is updated in real time based on real-time computing cluster performance metrics to generate a dynamic resource graph.

[0022] Preferably, the construction of the task-device matching cost matrix based on the dynamic resource graph and computational dependency graph specifically includes:

[0023] Based on the acquired dynamic resource map and computation dependency map, the computing power of the device and the computing requirements of the task are obtained, and the resource adaptability is calculated to evaluate the cost.

[0024] Based on the device’s current computing load and energy efficiency, assess the cost of allocating tasks to the device.

[0025] The network dependencies between subtasks are obtained based on the computation dependency graph, and the cost of network bandwidth and latency is evaluated based on data transmission.

[0026] The cost assessment results based on multiple factors are used to comprehensively evaluate the matching of each task and device, and a task-device matching cost matrix is ​​constructed.

[0027] Preferably, the step of optimizing the cost matrix based on a greedy algorithm to obtain the operator-level device allocation scheme and cross-device data routing path specifically includes:

[0028] Based on a greedy algorithm, select the operator with the minimum computational cost and the best match between resource requirements and device resource status from all tasks to be assigned, assign it to the most suitable device, and update the device's resource usage.

[0029] For each remaining operator, the device with the lowest current cost is selected for allocation based on the cost matrix. After each operator allocation, the cost matrix and the state of the device are updated.

[0030] Based on the dependency between the output data of each operator and the input data of the next operator, the cost of data transmission is calculated and the data transmission path is obtained.

[0031] Repeat the above steps to assign the remaining operators to the most suitable devices until all operators have been assigned, and obtain all assignment schemes and data routing paths.

[0032] Preferably, the step of distributing the tasks to be inferred to the inference device in operator-granularity slices based on the allocation scheme, and continuously monitoring the actual inference rate of the device during the execution of the sub-tasks, specifically includes:

[0033] Based on the obtained allocation scheme, the task to be inferred is sliced ​​according to operators, and the granularity of the operators is determined according to the computational dependencies of the tasks.

[0034] The slicing task is distributed to different devices for parallel execution, and the inference rate of each device is continuously monitored. The inference rate is the number of operators and the amount of data that the device can process per second.

[0035] Preferably, the dynamic reallocation mechanism triggered when the real-time inference rate of the target device is detected to be lower than a preset threshold, based on the actual inference rate of the device, specifically includes:

[0036] The device's inference rate is evaluated by comparing the actual inference rate of the device obtained in real time with the preset threshold.

[0037] If the device's inference rate is lower than the threshold, check the device's computing resource usage, device memory or video memory usage, and cross-device computing tasks to find the bottleneck cause of the low-speed device.

[0038] Determine whether a reallocation mechanism needs to be triggered based on the cause of the bottleneck.

[0039] Based on the redistribution mechanism, according to task dependencies and device load, some operators are migrated from low-speed devices to other devices with lighter loads, and resources are redistributed among devices.

[0040] Furthermore, a distributed inference acceleration system based on dynamic allocation of heterogeneous computing power is proposed, including:

[0041] Task receiving and splitting module: The task receiving and splitting module is used to receive the task to be reasoned, split it into multiple operator subtasks, and construct a computational dependency graph to ensure that the task is reasonably decomposed;

[0042] Computing Cluster Module: The computing cluster module is used to build a heterogeneous device computing cluster, collect and monitor the computing power type, video memory, network bandwidth, computing load and energy efficiency of each device in real time, and generate a dynamic resource map;

[0043] Cost Matrix Module: The cost matrix module evaluates the matching cost between tasks and devices based on dynamic resource graphs and computational dependency graphs, and generates a task-device matching cost matrix;

[0044] Optimization Allocation Module: The optimization allocation module optimizes the cost matrix based on a greedy algorithm, selects the optimal device for operator allocation, calculates the data routing path, and optimizes cross-device task scheduling;

[0045] Inference rate monitoring module: The inference rate monitoring module slices the task and distributes it to the device according to the operator granularity, while monitoring the inference rate of the device in real time to ensure efficient task execution;

[0046] The reassignment module compares the device's real-time inference rate with a preset threshold. When the rate is lower than the threshold, a reassignment mechanism is triggered to migrate tasks to devices with lighter loads.

[0047] Processor: The processor is used to handle the calculation process of each formula and the construction calculation process of each model.

[0048] Compared with the prior art, the advantages of the present invention are:

[0049] By monitoring the computing power, memory, and network bandwidth of each device in real time, a resource map is dynamically generated to ensure that each subtask can be executed on the most suitable device, thereby optimizing resource utilization. This method employs a greedy algorithm to optimize device allocation and dynamically adjusts task allocation based on the actual inference rate of each device, avoiding performance bottlenecks caused by device overload or insufficient resources. Specifically, during inference, when the inference rate of a device falls below a preset threshold, the system automatically triggers a reallocation mechanism, migrating tasks to lighter-loaded devices, ensuring efficient task execution and system stability. Furthermore, through task splitting, constructing a computation dependency graph, and optimizing cross-device data routing, this method effectively reduces data transmission latency and significantly improves the overall performance of inference tasks. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the method proposed in this invention;

[0051] Figure 2 This is a schematic diagram of the construction of the computational dependency graph proposed in this invention;

[0052] Figure 3 This is a schematic diagram of the dynamic resource map generation proposed in this invention;

[0053] Figure 4 This is a schematic diagram of the construction task device matching cost matrix proposed in this invention;

[0054] Figure 5 This is a schematic diagram of the device allocation scheme and routing path proposed in this invention;

[0055] Figure 6 This is a schematic diagram of the inference rate of the monitoring device proposed in this invention;

[0056] Figure 7 This is a schematic diagram of the dynamic reallocation triggering proposed in this invention. Detailed Implementation

[0057] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0058] A distributed inference acceleration system based on dynamic allocation of heterogeneous computing power includes:

[0059] Task receiving and splitting module: The task receiving and splitting module is used to receive the task to be reasoned, split it into multiple operator subtasks, and construct a computational dependency graph to ensure that the task is reasonably decomposed;

[0060] Computing Cluster Module: The computing cluster module is used to build a heterogeneous device computing cluster, collect and monitor the computing power type, video memory, network bandwidth, computing load and energy efficiency of each device in real time, and generate a dynamic resource map;

[0061] Cost Matrix Module: The cost matrix module evaluates the matching cost between tasks and devices based on dynamic resource graphs and computational dependency graphs, and generates a task-device matching cost matrix;

[0062] Optimization Allocation Module: The optimization allocation module optimizes the cost matrix based on a greedy algorithm, selects the optimal device for operator allocation, calculates the data routing path, and optimizes cross-device task scheduling;

[0063] Inference rate monitoring module: The inference rate monitoring module slices the task and distributes it to the device according to the operator granularity, while monitoring the inference rate of the device in real time to ensure efficient task execution;

[0064] The reassignment module compares the device's real-time inference rate with a preset threshold. When the rate is lower than the threshold, a reassignment mechanism is triggered to migrate tasks to devices with lighter loads.

[0065] Processor: The processor is used to handle the calculation process of each formula and the construction calculation process of each model.

[0066] See Figure 1 As shown, the distributed inference acceleration method based on dynamic allocation of heterogeneous computing power includes:

[0067] Step 1: Receive the task to be inferred, break it down into multiple operator subtasks and construct a computation dependency graph;

[0068] Step 2: Construct a computing cluster containing heterogeneous devices, collect data in real time on the computing power type, available video memory, network bandwidth, real-time computing load and unit energy consumption of each device, and generate a dynamic resource map;

[0069] Step 3: Construct a task-device matching cost matrix based on the dynamic resource graph and computational dependency graph;

[0070] Step 4: Optimize the cost matrix using a greedy algorithm to obtain the operator-level device allocation scheme and cross-device data routing paths;

[0071] Step 5: Based on the allocation scheme, distribute the tasks to be inferred to the inference device in operator-granularity slices, and continuously monitor the actual inference rate of the device during the execution of the sub-tasks;

[0072] Step 6: Based on the actual inference rate of the device, when the real-time inference rate of the target device is detected to be lower than the preset threshold, a dynamic reallocation mechanism is triggered.

[0073] See Figure 2 As shown, receiving the task to be inferred, breaking it down into multiple operator subtasks, and constructing a computational dependency graph specifically includes:

[0074] Based on the computational requirements of the task to be reasoned, the task is divided into multiple independent operators, each of which is a subtask;

[0075] Based on the data flow between operators, the dependencies between operators are obtained;

[0076] A computational dependency graph is constructed based on the obtained dependencies, with operators acting as nodes in the graph and edges representing the data dependencies between operators.

[0077] Specifically, based on the computational requirements of the task, each computational operation is broken down into an independent operator, and each operator performs a specific computational step. For example, a linear layer can be broken down into matrix multiplication and addition operations, and a convolutional layer can be broken down into convolution operations and the application of activation functions.

[0078] After identifying the operators, it is necessary to analyze the data flow between them. The output of one operator is usually used as the input of another operator; therefore, the dependencies between operators must be determined based on the data flow relationships. Based on these dependencies, a computational dependency graph is constructed, where each operator is a node in the graph. If operator S... i The output is operator S j The input, then the graph starts from node S i Pointing to node S j That is, add a line from S to the graph. i To S j After constructing the computation dependency graph with directed edges, the graph is topologically sorted to ensure the correct computation order.

[0079] See Figure 3 As shown, a computing cluster containing heterogeneous devices is constructed. Real-time data collection of each device's computing power type, available video memory, network bandwidth, real-time computing load, and unit energy consumption indicators is used to generate a dynamic resource map. Specifically, this includes:

[0080] Construct a computing cluster containing heterogeneous devices, including CPUs, GPUs, FPGAs, TPUs, and NPUs;

[0081] Obtain computing cluster performance metrics, including computing power type, available video memory, network bandwidth, real-time computing load, and unit energy consumption metrics;

[0082] Based on the acquired data, a resource map is designed, where each node represents a device. The size, color, and shape of the node indicate the device's load, available memory, and network bandwidth information.

[0083] The graph data is updated in real time based on real-time computing cluster performance metrics to generate a dynamic resource graph.

[0084] Specifically, a heterogeneous computing cluster is built based on different types of computing devices, each with different computing capabilities, storage capabilities, and network characteristics, including:

[0085] CPU: Central Processing Unit, suitable for serial computing, supports multi-core parallelism;

[0086] GPU: Graphics processing unit, suitable for large-scale parallel computing, typically used for deep learning and graphics rendering;

[0087] FPGA: Field Programmable Gate Array, suitable for customized computing tasks, providing high-performance and low-power solutions;

[0088] TPU: Tensor Processing Unit, designed specifically for machine learning tasks, especially for deep learning inference;

[0089] NPU: Neural Processing Unit, specifically optimized for accelerating inference in neural network models;

[0090] For each device in the cluster, key performance indicators are collected in real time, and a resource graph is designed and constructed. Each node represents a heterogeneous device, and the size, color, and shape of the node represent the real-time status of the device through different attributes.

[0091] The latest performance metrics of each device in the computing cluster are collected periodically, and the graph is updated based on this data to obtain a dynamic resource graph.

[0092] See Figure 4 As shown, the construction of the task-device matching cost matrix based on the dynamic resource graph and computational dependency graph specifically includes:

[0093] Based on the acquired dynamic resource map and computation dependency map, the computing power of the device and the computing requirements of the task are obtained, and the resource adaptability is calculated to evaluate the cost.

[0094] Based on the device’s current computing load and energy efficiency, assess the cost of allocating tasks to the device.

[0095] The network dependencies between subtasks are obtained based on the computation dependency graph, and the cost of network bandwidth and latency is evaluated based on data transmission.

[0096] The cost assessment results based on multiple factors are used to comprehensively evaluate the matching of each task and device, and a task-device matching cost matrix is ​​constructed.

[0097] Specifically, the computing power of each device is obtained based on the number of floating-point operations per second, and the computing requirements of the task are obtained by analyzing the computing dependency graph. The resource adaptability is used to assess whether the task can be executed smoothly on the device. It is usually based on the ratio of the device's computing power to the task's computing requirements. The higher the adaptability, the better the adaptability of the task to the device.

[0098] The computing load of a device is determined by the proportion of computing resources currently used to the total resources. Energy efficiency represents the energy consumed per unit of computing power. The cost of a device comprehensively considers both computing load and energy efficiency, and the formula is as follows:

[0099] D d =L cost +E cost =(L d ×C d )+(E d ×C d );

[0100] Among them, L cost For the cost of load, E cost For the cost of energy efficiency, L d To calculate the load, C d E represents the computing power of device d. d For unit energy efficiency, D d For the cost of equipment;

[0101] The computation dependency graph describes the data transmission relationships between various subtasks of a task. For data transmission between devices, the higher the network bandwidth, the faster the data transmission speed, the lower the latency, and the lower the cost.

[0102] The computational cost, load cost, energy efficiency cost, network bandwidth and latency cost of the equipment are comprehensively evaluated to obtain the comprehensive cost of task allocation to the equipment. By calculating the comprehensive cost of all tasks and equipment, a task-equipment matching cost matrix is ​​constructed. The dimension of the matrix is ​​the product of the number of equipment and the number of tasks. Each row represents the cost of a task to all equipment, and the column represents the cost of all equipment to the task.

[0103] See Figure 5 As shown, the optimization solution of the cost matrix based on the greedy algorithm to obtain the operator-level device allocation scheme and cross-device data routing path specifically includes:

[0104] Based on a greedy algorithm, select the operator with the minimum computational cost and the best match between resource requirements and device resource status from all tasks to be assigned, assign it to the most suitable device, and update the device's resource usage.

[0105] For each remaining operator, the device with the lowest current cost is selected for allocation based on the cost matrix. After each operator allocation, the cost matrix and the state of the device are updated.

[0106] Based on the dependency between the output data of each operator and the input data of the next operator, the cost of data transmission is calculated and the data transmission path is obtained.

[0107] Repeat the above steps to assign the remaining operators to the most suitable devices until all operators have been assigned, and obtain all assignment schemes and data routing paths.

[0108] Specifically, from all the tasks to be assigned, select the operator with the lowest computational cost and the best match between resource requirements and device resource status for assignment. Once the device resource status changes, the cost matrix needs to be updated according to the new resource status, and the cost of each device and task combination needs to be recalculated.

[0109] During task allocation, data transfer occurs between the outputs and inputs of operators. The cost of data transfer mainly involves bandwidth and latency, and the formula for the cost of data transfer is as follows:

[0110]

[0111] Among them, T cost For the cost of data transmission, D trans For the amount of data transmitted, B net For transmission bandwidth, T latency For delay;

[0112] For the remaining operators, continue to select the device with the lowest current cost based on the cost matrix for allocation. After each operator allocation, the resource usage of the device needs to be updated. Repeat the above steps until all operators have been allocated. Each time, the device with the lowest cost is selected for allocation, and the resource status and cost matrix are updated in real time.

[0113] By executing the greedy algorithm process described above multiple times, all operators are eventually assigned to the most suitable devices, generating a complete task device matching scheme and data transmission path.

[0114] See Figure 6 As shown, the inference task is distributed to the inference device in operator-granularity slices based on the allocation scheme, and the actual inference rate of the device is continuously monitored during the execution of the subtasks. Specifically, this includes:

[0115] Based on the obtained allocation scheme, the task to be inferred is sliced ​​according to operators, and the granularity of the operators is determined according to the computational dependencies of the tasks.

[0116] The slicing task is distributed to different devices for parallel execution, and the inference rate of each device is continuously monitored. The inference rate is the number of operators and the amount of data that the device can process per second.

[0117] See Figure 7 As shown, based on the actual inference rate of the device, when the real-time inference rate of the target device is detected to be lower than a preset threshold, the dynamic reallocation mechanism is triggered, specifically including:

[0118] The device's inference rate is evaluated by comparing the actual inference rate of the device obtained in real time with the preset threshold.

[0119] If the device's inference rate is lower than the threshold, check the device's computing resource usage, device memory or video memory usage, and cross-device computing tasks to find the bottleneck cause of the low-speed device.

[0120] Determine whether a reallocation mechanism needs to be triggered based on the cause of the bottleneck.

[0121] Based on the redistribution mechanism, according to task dependencies and device load, some operators are migrated from low-speed devices to other devices with lighter loads, and resources are redistributed among devices.

[0122] Specifically, a device's inference rate represents the number of inference tasks it can process per unit of time, usually expressed as the number of inference tasks processed or the number of samples processed. The device's inference rate is obtained through real-time monitoring and statistics. If the device's inference rate is below a threshold, it is necessary to check the device's computing resource usage, memory or video memory usage, and whether task dependencies span multiple devices.

[0123] If the inference rate falls below a threshold and a bottleneck is identified, such as device computing resource overload, excessive memory usage, or cross-device computing bottlenecks, a reallocation mechanism is triggered. The goal of the reallocation mechanism is to migrate some operators from currently heavily loaded devices to less loaded devices, thereby reducing the device load.

[0124] Based on task dependencies and device load, some operators on low-speed devices are migrated to other devices with lighter loads. When selecting operators and target devices for migration, it is necessary to consider whether the target device has sufficient computing resources, memory, and bandwidth to receive tasks. The load of the migrated device will not exceed its maximum carrying capacity. At the same time as operator migration, resources between devices need to be reallocated and the resource occupancy status of each device needs to be adjusted.

[0125] After the reallocation is completed, the inference rate and task allocation effectiveness of the devices are reassessed. In actual operation, the inference rate and device status are dynamic. It is necessary to continuously monitor the device's inference rate and load. If a new bottleneck appears, the reallocation mechanism should be triggered again to ensure optimal utilization of system resources.

[0126] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0127] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed inference acceleration method based on dynamic allocation of heterogeneous computing power, characterized in that, The application comprises the following steps: receiving a task to be inferred, splitting the task into multiple operator sub-tasks and constructing a calculation dependency graph; constructing a calculation cluster comprising heterogeneous devices, collecting the computing power type, available video memory, network bandwidth, real-time calculation load and unit energy consumption index of each device in real time, and generating a dynamic resource atlas; based on the dynamic resource atlas and the calculation dependency graph, constructing a task-device matching cost matrix; based on the greedy algorithm, optimizing and solving the cost matrix to obtain an operator-level device allocation scheme and a cross-device data routing path; based on the allocation scheme, slicing and distributing the task to be inferred to the inference device in the operator granularity, and continuously monitoring the actual inference rate of the device during the execution of the sub-tasks; based on the actual inference rate of the device, when it is detected that the real-time inference rate of the target device is lower than the preset threshold, triggering a dynamic reallocation mechanism.

2. The method of claim 1, wherein, The step of receiving a task to be inferred, splitting the task into multiple operator sub-tasks and constructing a calculation dependency graph specifically comprises the following steps: based on the calculation requirement of the task to be inferred, splitting the task into multiple independent operators, each operator being a sub-task; based on the data flow between the operators, obtaining the dependency relationship between the operators; based on the obtained dependency relationship, constructing a calculation dependency graph, with the operators as nodes in the graph and the edges as the data dependency between the operators.

3. The method of claim 1, wherein, The step of constructing a calculation cluster comprising heterogeneous devices, collecting the computing power type, available video memory, network bandwidth, real-time calculation load and unit energy consumption index of each device in real time, and generating a dynamic resource atlas specifically comprises the following steps: constructing a calculation cluster comprising heterogeneous devices, wherein the heterogeneous devices comprise CPU, GPU, FPGA, TPU and NPU; obtaining the performance indicators of the calculation cluster, including the computing power type, available video memory, network bandwidth, real-time calculation load and unit energy consumption index; based on the obtained data, designing a resource atlas, with each node representing a device, and representing the load, available memory and network bandwidth information of the device through the size, color and shape of the node; based on the real-time performance indicators of the calculation cluster, updating the atlas data in real time to generate a dynamic resource atlas.

4. The method of claim 1, wherein, The step of constructing a task-device matching cost matrix based on the dynamic resource atlas and the calculation dependency graph specifically comprises the following steps: based on the obtained dynamic resource atlas and the calculation dependency graph, obtaining the computing power of the device and the calculation requirement of the task, and calculating the resource adaptation degree for cost evaluation; based on the current calculation load and energy efficiency of the device, evaluating the cost of assigning the task to the device; based on the network dependency between the sub-tasks obtained from the calculation dependency graph, evaluating the cost according to the network bandwidth and delay of data transmission; based on the cost evaluation results of multiple factors, comprehensively evaluating the matching of each task and device to construct a task-device matching cost matrix.

5. The method of claim 1, wherein, The step of optimizing and solving the cost matrix based on the greedy algorithm to obtain an operator-level device allocation scheme and a cross-device data routing path specifically comprises the following steps: based on the greedy algorithm, selecting an operator with the smallest calculation cost and the most matched resource requirement and device resource state from all the tasks to be allocated, and updating the resource occupation status of the device; For each remaining operator, according to the cost matrix, the device with the minimum current cost is selected for allocation, and after each operator allocation, the cost matrix and the state of the device are updated; Based on the dependency relationship between the output data of each operator and the input data of the next operator, the cost of data transmission is calculated, and the data transmission path is obtained; Repeat the above steps to allocate the remaining operators to the most suitable devices until all operators are allocated, and obtain all allocation schemes and data routing paths.

6. The method of claim 1, wherein, The task to be reasoned is sliced into operators based on the obtained allocation scheme, and the operator granularity is determined according to the calculation dependency of the task. The sliced tasks are distributed to different devices for parallel execution, and the reasoning rate of each device is continuously monitored, which is the number of operators and data volume that the device can process per second. Based on the actual reasoning rate of the device, when the real-time reasoning rate of the target device is detected to be lower than the preset threshold, the dynamic reallocation mechanism is triggered.

7. The method of claim 1, wherein, Based on the real-time actual reasoning rate of the device and the set preset threshold, the device reasoning rate is evaluated. If the reasoning rate of the device is lower than the threshold, the calculation resource occupation of the device, the memory or video memory usage of the device, and the cross-device calculation task are checked to obtain the bottleneck reason of the low-rate device. Based on the bottleneck reason, it is judged whether the reallocation mechanism needs to be triggered. Based on the reallocation mechanism, according to the task dependency relationship and the device load situation, part of the operators are migrated from the low-rate device to other devices with lighter load, and the resources between devices are reallocated. It includes:

8. A distributed inference acceleration system based on dynamic allocation of heterogeneous computing power, configured to implement the method of any one of claims 1-7, characterized in that, Task receiving and splitting module: the task receiving and splitting module is used to receive the task to be reasoned, split it into multiple operator sub-tasks, and construct a calculation dependency graph to ensure that the task is reasonably decomposed; Computing cluster module: the computing cluster module is used to construct a heterogeneous device computing cluster, and real-time collect and monitor the computing power type, video memory, network bandwidth, computing load and energy efficiency of each device to generate a dynamic resource map; Cost matrix module: the cost matrix module evaluates the matching cost between the task and the device based on the dynamic resource map and the calculation dependency graph, and generates a task-device matching cost matrix; Optimized allocation module: the optimized allocation module optimizes the cost matrix based on the greedy algorithm, selects the optimal device for operator allocation, and calculates the data routing path to optimize the cross-device task scheduling; Inference rate monitoring module: the inference rate monitoring module slices the task by operator granularity and distributes it to the device, while monitoring the inference rate of the device in real time to ensure efficient task execution; Reallocation module: the reallocation module compares the real-time inference rate of the device with the preset threshold, and when the rate is lower than the threshold, triggers the reallocation mechanism to migrate the task to a device with lighter load; Processor: the processor is used to process the calculation process of each formula and the construction calculation process of each model. ​

Images