End-side multi-model collaborative reasoning method, end-side equipment and storage medium

Abstract

The invention discloses an end-side multi-model collaborative reasoning method, end-side equipment and a storage medium, and relates to the technical field of computers.The hardware resource state data of the end-side equipment are dynamically collected, and the hardware resource state data at least comprise the computing power utilization rate, the memory occupancy rate and the operation power consumption in all heterogeneous computing units; obtaining static metadata and dynamic reasoning performance data of each AI model; based on the hardware resource state data, the static metadata and the dynamic reasoning performance data, computing resources of heterogeneous computing units are dynamically allocated to all the AI models, the reasoning priority of all the AI models is determined, and a parallel strategy is executed to generate a resource scheduling strategy; and according to the resource scheduling strategy and the model dependency relationship, constructing and executing a system-level reasoning task assembly line to complete collaborative reasoning. The resource utilization rate and the reasoning efficiency of an end-side multi-model reasoning scheme can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to edge-side multi-model collaborative reasoning methods, edge-side devices, and storage media. Background Technology

[0002] With the rapid development of artificial intelligence technology, edge devices (such as smartphones, IoT terminals, and autonomous driving edge computing units) are undertaking increasingly complex intelligent tasks. To meet diverse application needs, a single edge device typically needs to deploy and collaboratively run multiple different AI models (e.g., simultaneously running models for object detection, speech recognition, and semantic understanding). These models usually have different computational graph structures, operator types, and resource requirements, and may have data dependencies on each other.

[0003] However, the computing resources of edge devices (such as heterogeneous computing units like CPUs, GPUs, and NPUs) are severely limited in terms of computing power, memory, and power consumption. Currently, most common multi-model inference solutions employ static or semi-static resource allocation and scheduling strategies. This means that before deployment, fixed computing resources are pre-allocated to each model based on experience or offline analysis, and a fixed execution order or parallel mode is set. Such solutions have significant shortcomings: First, resource allocation is rigid and cannot adapt to dynamic loads. The operating environment of edge devices is complex and variable (such as user operations, network conditions, background tasks, etc.), causing hardware resources (such as CPU / GPU / NPU utilization, memory usage, power consumption) and the model's own inference load (such as latency, throughput) to fluctuate dynamically. Static allocation strategies cannot be dynamically adjusted according to real-time status, resulting in resource contention and latency during high loads, and resource idleness and waste during low loads.

[0004] Secondly, scheduling strategies lack global collaborative optimization. Existing solutions often consider the performance of a single model in isolation or schedule based solely on simple priority rules, failing to fully consider the dependencies between multiple models, the specificity of heterogeneous computing units, and the compatibility at the model operator level. This makes it impossible to build a globally optimal inference pipeline, hindering cross-model operator fusion, data flow optimization, and efficient parallel execution on heterogeneous hardware.

[0005] Therefore, improving the resource utilization and inference efficiency of edge-side multi-model inference schemes on resource-constrained edge devices has become an urgent technical problem to be solved. Summary of the Invention

[0006] The main purpose of this application is to provide a multi-model collaborative inference method, a device, and a storage medium for edge devices, aiming to solve the technical problems of low resource utilization and insufficient inference efficiency in edge multi-model inference schemes.

[0007] To achieve the above objectives, this application proposes an edge-side multi-model collaborative inference method, applied to an edge-side device containing multiple heterogeneous computing units, wherein multiple AI models are deployed on the edge-side device, characterized by comprising: The hardware resource status data of the end-side device is dynamically collected. The hardware resource status data includes at least the computing power utilization rate, memory occupancy rate and operating power consumption of each heterogeneous computing unit. Obtain static metadata and dynamic inference performance data for each AI model. The static metadata includes the computation graph structure, operator type, and input / output tensor dimensions parsed from the AI ​​model. The dynamic inference performance data includes the actual latency, throughput, and resource utilization rate on each heterogeneous computing unit during the model inference process of the AI ​​model. Based on the hardware resource status data, the static metadata, and the dynamic inference performance data, computing resources of the heterogeneous computing units are dynamically allocated to each AI model, and the inference priority and execution parallel strategy of each AI model are determined to generate a resource scheduling strategy. The execution parallel strategy is used to indicate whether models without dependencies are executed in parallel and whether a single model is computed in parallel on multiple computing units. The inference priority and execution parallel strategy need to be decided in combination with the model dependencies between each AI model. Based on the resource scheduling strategy and the model dependencies, a system-level inference task pipeline is constructed and executed to complete collaborative inference. The construction of the system-level inference task pipeline includes: determining the execution order based on the model dependencies, constructing a directed acyclic computation graph representing the data flow between models, and performing operator fusion on adjacent and operator-compatible model nodes in the directed acyclic computation graph.

[0008] In one embodiment, after the step of constructing and executing a system-level inference task pipeline to complete collaborative inference based on the resource scheduling strategy and the model dependencies, the method further includes: The system collects performance metrics during the execution of the system-level inference task pipeline and feeds these performance metrics back to update the dynamic inference performance data in order to optimize the resource scheduling strategy and inference task pipeline for the next round.

[0009] In one embodiment, the step of dynamically collecting the hardware resource status data of the end-side device includes: The hardware resource status data is collected at a dynamically adjustable collection period; The collection cycle is adjusted based on the current system load or the urgency of the AI ​​model's inference task. When the system load is higher than a preset threshold or there are tasks with an inference task urgency greater than a preset value, the collection cycle is shortened to increase the monitoring frequency. When the system load is lower than a preset threshold and there are no tasks with an inference task urgency greater than a preset value, the collection cycle is extended to reduce system overhead.

[0010] In one embodiment, the plurality of heterogeneous computing units include at least two of CPU, GPU, and NPU. The step of dynamically allocating computing resources to each of the AI ​​models based on the hardware resource status data, the static metadata, and the dynamic inference performance data includes: Establish and maintain a global resource pool to perform unified abstraction, quantification, and state management of at least two types of computing resources, namely CPU, GPU, and NPU. Based on the hardware resource status data, the static metadata, and the dynamic inference performance data, and combined with the real-time available resources of the global resource pool, computing resources are dynamically allocated to each of the AI ​​models.

[0011] In one embodiment, the step of determining the inference priority of each model and the execution parallel strategy includes: Based on the historical sequence of the dynamic inference performance data, an adaptive load balancing algorithm is used to predict the changes in computing power demand of each AI model in the next time window. By combining the predicted changes in computing power demand, the model dependencies, and the preset task urgency indicators, the inference priority of each model is calculated and dynamically adjusted.

[0012] In one embodiment, the step of feeding back the performance metrics to update the dynamic inference performance data to optimize the resource scheduling strategy for the next round includes: The actual latency and throughput collected are compared with the expected performance under the current resource scheduling strategy to generate prediction error; Based on the prediction error, an online learning mechanism is used to adjust the internal weight parameters of the adaptive load balancing algorithm used to predict changes in computing power demand.

[0013] In one embodiment, the method further includes a fault recovery mechanism: Real-time monitoring of the health status of each heterogeneous computing unit; When a fault or severe performance degradation is detected in the heterogeneous computing unit where the computing resources allocated to execute any AI model are located, the affected model inference task is migrated to other available heterogeneous computing units for continued execution based on the real-time available resources of the global resource pool, or the affected model inference task reduces its inference accuracy to reduce computing power requirements.

[0014] In one embodiment, the step of operator fusion of adjacent and operator-compatible model nodes in the directed acyclic computation graph includes: Determine whether the output operator of the upstream model node in the directed acyclic computation graph matches the input operator of the downstream model node in the directed acyclic computation graph in terms of data precision, data dimension, and computation type. If the matching degree exceeds the preset fusion threshold, the output operator and the input operator are fused into a composite operator. This composite operator is executed continuously within a computing unit to reduce the number of times intermediate data is moved between memory and different heterogeneous computing units.

[0015] In one embodiment, the step of constructing and executing the system-level inference task pipeline further includes: Based on the resource scheduling strategy, the model dependencies, the model nodes in the directed acyclic computation graph, and the fused operator types, the operators undertaking computational tasks in each AI model are mapped to the heterogeneous computing units corresponding to the global resource pool. Pipeline parallel technology is then used to execute the system-level inference task pipeline, enabling different AI models at different stages in the directed acyclic computation graph to process different batches or different parts of data simultaneously, thereby improving overall hardware utilization and system throughput.

[0016] In one embodiment, the method further includes: A visual monitoring interface is provided to display in real time the hardware resource status data, the dynamic inference performance data of each AI model, the inference priority, the resource scheduling strategy, and the execution status of the system-level inference task pipeline; The system receives instructions for external manual intervention through the visual monitoring interface. These manual interventions include: adjusting the inference priority of the AI ​​model, triggering or terminating operator fusion, and reallocating computing resources to the AI ​​model.

[0017] Furthermore, to achieve the above objectives, this application also proposes an edge-side multi-model collaborative reasoning device, applied to an edge-side device, wherein the edge-side device includes a first-side camera, and the edge-side multi-model collaborative reasoning device includes: The depth information acquisition module is used to acquire first-side depth information of the target scene, wherein the first-side depth information is the depth distribution of the target scene under the view of the first-side camera; The spatial transformation relationship determination module is used to determine the spatial transformation relationship between the first side camera and the second side camera based on preset camera spatial position relationship parameters; The edge-side multi-model collaborative reasoning module is used to infer the second-side depth information of the target scene based on the spatial transformation relationship and the first-side edge-side multi-model collaborative reasoning.

[0018] In addition, to achieve the above objectives, this application also proposes an edge device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the edge multi-model collaborative reasoning method as described above.

[0019] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the edge-side multi-model collaborative reasoning method described above.

[0020] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the edge-side multi-model collaborative reasoning method described above.

[0021] This application provides an edge-side multi-model collaborative inference method, applied to an edge device containing multiple heterogeneous computing units. The edge device deploys multiple AI models. This application constructs a closed-loop optimization process from perception and decision-making to execution, specifically including: First, dynamic data collection and acquisition are performed: real-time dynamic collection of hardware resource status data of edge devices (such as computing power utilization, memory usage, and power consumption of each processor), and acquisition of static metadata (such as computation graph structure and operator type) and dynamic inference performance data (such as actual latency, throughput, and resource usage) of each AI model. Next, dependency-aware dynamic scheduling is performed: based on the aforementioned hardware resource status data, static metadata, and dynamic inference performance data, computing resources of heterogeneous computing units are dynamically allocated to each AI model, and the inference priority and execution parallelism strategy of each model are determined, thereby generating a resource scheduling strategy. The execution parallelism strategy guides parallel execution between dependent models and parallel computation within a single model, and the decision-making process for inference priority and execution parallelism strategy must consider the model dependencies between each AI model to ensure that the scheduling results conform to the logic of the overall workflow. Then, a system-level pipeline is constructed and executed: based on the generated resource scheduling strategy and the dependencies between models, a system-level inference task pipeline is constructed and executed. The key lies in determining the execution order based on model dependencies, constructing a directed acyclic computation graph representing the data flow between models, and performing operator fusion on adjacent and operator-compatible model nodes in this graph to eliminate unnecessary data transfer, ultimately forming a highly efficient and collaborative execution pipeline. Based on the above closed-loop optimization process, dynamic, collaborative, and efficient inference of multiple AI models can be effectively achieved on resource-constrained edge devices. The system status and model requirements are perceived in real time, and heterogeneous resources are intelligently allocated, execution strategies are determined, and optimized pipelines are constructed accordingly. This solves the technical problems of low resource utilization and insufficient inference efficiency in edge-side multi-model inference schemes. Compared with related technologies, this invention has the following beneficial effects: 1. Achieved dynamic adaptive resource scheduling: By collecting hardware and model data in real time and dynamically allocating resources and adjusting scheduling strategies based on this data, the system can keep up with load changes, effectively alleviate resource competition, reduce resource idleness, and significantly improve the overall utilization rate of heterogeneous computing resources.

[0022] 2. Achieved global optimization for multi-model collaboration: By comprehensively considering model dependencies, heterogeneous hardware characteristics, and operator compatibility, global decisions are made from resource allocation to pipeline construction. In particular, the introduction of dependency-based operator fusion reduces unnecessary data movement, optimizes data flow, and thus improves end-to-end inference efficiency.

[0023] 3. Enhanced system applicability and efficiency in complex scenarios: The method proposed in this application can adapt to the dynamic and ever-changing environment of edge devices, intelligently achieving a balance between performance, resources, and power consumption. Through dynamic and collaborative scheduling and optimization, it can still ensure the efficient and stable execution of multi-AI model collaborative inference tasks under strictly limited resource conditions. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the first embodiment of the multi-model collaborative reasoning method for the end-side of this application. Figure 2This is a flowchart illustrating the second embodiment of the multi-model collaborative reasoning method for the end-side of this application. Figure 3 This is a flowchart illustrating the third embodiment of the multi-model collaborative reasoning method for the end-side of this application. Figure 4 This is a schematic diagram of the system architecture of the edge-side multi-model collaborative reasoning method according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the end-side multi-model collaborative reasoning method in the embodiments of this application.

[0027] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0028] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0029] It should be noted that the executing entity in this embodiment can be an edge device with data processing, network communication, and program execution functions. This edge device includes multiple heterogeneous computing units and deploys multiple AI models. The type of edge device includes, for example, wearable devices, tablets, personal computers, mobile phones, vehicles, smart cameras, or smart door controllers, or any edge device capable of performing the aforementioned functions. Wearable devices can include, but are not limited to, smart bracelets, smartwatches, smart rings, smart headphones, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof. The following description uses an edge device as an example to illustrate this embodiment and the subsequent embodiments.

[0030] For ease of understanding, the following explains some key terms in this embodiment: Edge devices refer to devices located at the network edge that possess certain computing, storage, and communication capabilities, such as smartphones, IoT terminals, and autonomous driving edge computing units. These devices are typically resource-constrained but require real-time data processing and execution of artificial intelligence tasks.

[0031] Heterogeneous computing units refer to different types of computing processors integrated within an edge device, such as central processing units (CPUs), graphics processing units (GPUs), and neural network processors (NPUs). These computing units differ in architecture, instruction sets, computing power, and power consumption characteristics, and are suitable for handling different types of computing tasks.

[0032] AI models are algorithmic models that have been trained to perform specific artificial intelligence tasks, such as object detection models, speech recognition models, and semantic understanding models. On edge devices, multiple AI models typically need to be deployed and run collaboratively to meet diverse application requirements.

[0033] Hardware resource status data refers to data reflecting the current hardware resource usage of edge devices, which includes at least the computing power utilization, memory usage, and power consumption of each heterogeneous computing unit. This data is used to assess the device's load and resource availability in real time.

[0034] Static metadata refers to the inherent attribute information of an AI model that can be obtained before deployment. This includes the computational graph structure, operator types, and input / output tensor dimensions parsed from the AI ​​model. This data describes the model's internal structure and data processing characteristics.

[0035] Dynamic inference performance data refers to the performance metrics generated by AI models during actual inference, including the model's actual latency, throughput, and resource utilization on various heterogeneous computing units. These data reflect the model's actual operating performance under specific hardware and load conditions.

[0036] A computation graph structure is a graphical representation of the computational operations and data flow within an AI model. This structure consists of a series of operator nodes and edges connecting these nodes, describing the model's computational logic.

[0037] Operator type refers to the basic unit in an AI model that performs specific computational operations, such as convolution operators, pooling operators, activation operators, or composite operators with combined properties. Different operators have different computational characteristics and resource requirements.

[0038] The input-output tensor dimension refers to the shape and size of the input and output data of an AI model. This dimensional information is crucial for determining data transfer and memory allocation.

[0039] Actual latency refers to the time required for an AI model to complete one inference task. Throughput refers to the number of inference tasks that an AI model can process per unit of time. Resource utilization refers to the proportion of computing power, memory, and other resources consumed by the AI ​​model during inference on specific heterogeneous computing units.

[0040] Inference priority refers to the execution order or importance level assigned to each AI model. Models with higher priority will receive computing resources or be executed first.

[0041] Parallel execution strategy refers to the way to guide AI model inference tasks to be executed in parallel. It is used to indicate whether models with no dependencies can be executed at the same time, and whether a single model can be computed in parallel on multiple computing units.

[0042] Resource scheduling strategy refers to a comprehensive scheme that allocates computing resources, determines inference priority, and executes parallel strategies for each AI model based on hardware resource status, model characteristics, and performance requirements.

[0043] Model dependency refers to the relationship between multiple AI models in terms of data input / output or logical execution order. For example, the output of one model may serve as the input of another model.

[0044] A system-level inference pipeline refers to organizing the inference tasks of multiple AI models into a continuous and efficient execution sequence to achieve collaborative inference. This pipeline improves overall system performance by optimizing data flow and parallel execution.

[0045] A directed acyclic computation graph (DAG) is a special type of computation graph where nodes represent computational tasks or data, edges represent data flow or task dependencies, and there are no cycles in the graph. This graph is used to represent dependencies between models and data flow.

[0046] Operator fusion refers to combining multiple adjacent and operator-compatible computational operators in a directed acyclic computation graph into a single composite operator. This operation aims to reduce intermediate data transfer and memory access overhead, thereby improving execution efficiency.

[0047] This embodiment provides an edge-side multi-model collaborative inference method, which is applied to an edge device containing multiple heterogeneous computing units and deploying multiple AI models. This method achieves dynamic, collaborative, and efficient inference of multiple AI models through a series of steps.

[0048] Based on this, this application provides an edge-side multi-model collaborative inference method. This edge-side multi-model collaborative inference method is applied to an edge device containing multiple heterogeneous computing units, on which multiple AI models are deployed. Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multi-model collaborative reasoning method for the end-side of this application.

[0049] In this embodiment, the multi-model collaborative reasoning method includes steps S100~S400: Step S100: Dynamically collect hardware resource status data of the end-side device. The hardware resource status data includes at least the computing power utilization rate, memory occupancy rate and operating power consumption of each heterogeneous computing unit. Step S200: Obtain static metadata and dynamic inference performance data for each AI model; The static metadata includes the computation graph structure, operator type, and input / output tensor dimensions parsed from the AI ​​model, while the dynamic inference performance data includes the actual latency, throughput, and resource utilization on each heterogeneous computing unit during the AI ​​model's inference process.

[0050] Step S300: Based on hardware resource status data, static metadata and dynamic inference performance data, dynamically allocate computing resources of heterogeneous computing units to each AI model, and determine the inference priority and execution parallel strategy of each AI model to generate a resource scheduling strategy. Among them, the execution parallel strategy is used to indicate whether models with no dependencies are executed in parallel, and whether a single model is computed in parallel on multiple computing units. The inference priority and execution parallel strategy need to be decided in combination with the model dependencies between each AI model.

[0051] Step S400: Based on the resource scheduling strategy and model dependencies, construct and execute a system-level inference task pipeline to complete collaborative inference.

[0052] The construction of a system-level inference task pipeline includes: determining the execution order based on model dependencies, constructing a directed acyclic computation graph representing the data flow between models, and performing operator fusion on adjacent and operator-compatible model nodes in the directed acyclic computation graph.

[0053] In this embodiment, the method includes dynamically collecting hardware resource status data of the edge device. This hardware resource status data includes at least the computing power utilization, memory usage, and power consumption of each heterogeneous computing unit. For example, a monitoring agent can be deployed on the edge device to periodically read hardware performance counters or sensor data provided by the operating system at fixed time intervals (e.g., once per second) to obtain the current load, memory usage, and real-time power consumption of computing units such as the CPU, GPU, and NPU. This data is collected and stored for subsequent resource scheduling decisions. The hardware resource status data of the edge device can be obtained through a dynamically adjustable collection period via a hardware driver interface or operating system kernel interface, wherein the collection period can be dynamically adjusted within a range of 10 milliseconds to 3 seconds based on the current overall load of the edge device.

[0054] Secondly, this method acquires static metadata and dynamic inference performance data for each AI model. The static metadata includes the computational graph structure, operator types, and input / output tensor dimensions parsed from the AI ​​model. For example, before deploying the AI ​​model, offline analysis of the model file can be performed using model parsing tools to extract its internal computational operation sequence, the type of each operation (such as convolution, pooling, activation, etc.), and the tensor shape information of the data input and output. The dynamic inference performance data includes the actual latency, throughput, and resource utilization on the heterogeneous computing unit during the model inference process. For example, a one-time performance benchmark test can be performed during the model's first run to record the average latency, maximum throughput, and average resource utilization of CPU, GPU, NPU, etc., when executed on different heterogeneous computing units. This static and dynamic data provides fundamental information for subsequent resource allocation and scheduling.

[0055] Furthermore, based on the hardware resource status data, the static metadata, and the dynamic inference performance data, computing resources of the heterogeneous computing units are dynamically allocated to each AI model, and the inference priority and execution parallelism strategy of each AI model are determined to generate a resource scheduling strategy. The execution parallelism strategy indicates whether models without dependencies are executed in parallel, and whether a single model is computed in parallel on multiple computing units. The inference priority and execution parallelism strategy need to be decided in conjunction with the model dependencies between the AI ​​models. Specifically, resource allocation can be performed using a rule-based heuristic algorithm based on the currently collected hardware resource status data, combined with the static metadata of the AI ​​models (such as model size and operator complexity) and pre-acquired dynamic inference performance data (such as the model's performance on different hardware). For example, computationally intensive models can be allocated to GPUs or NPUs, and control logic or lightweight models can be allocated to CPUs. Inference priorities can be simply sorted according to preset business importance or the arrival order of model input data. The execution parallelism strategy can make a preliminary judgment based on whether there are data dependencies between models; if there are no dependencies, parallel execution is allowed, and a single model is executed on a single computing unit by default. This leads to the generation of a preliminary resource scheduling strategy.

[0056] Finally, based on the resource scheduling strategy and model dependencies, a system-level inference task pipeline is constructed and executed to complete collaborative inference. Constructing this system-level inference task pipeline includes: determining the execution order based on model dependencies, building a directed acyclic computation graph (DAG) representing the data flow between models, and performing operator fusion on adjacent and operator-compatible model nodes within this DAG. Specifically, the input-output relationships between various AI models can be analyzed first to identify which models' outputs are inputs to other models, thus establishing a dependency graph between models. Based on this dependency graph, the topological execution order of the models can be determined. Subsequently, these models and their dependencies are abstracted into nodes and edges in the DAG. During the construction of this graph, simple operator fusion can be performed on adjacent model nodes whose operator types are compatible in terms of basic data types and operational semantics. For example, two consecutive linear layers or activation functions can be merged into a composite operator to reduce intermediate data read / write operations in memory. This fused computation graph is then submitted to heterogeneous computing units for execution, thereby completing the multi-model collaborative inference task.

[0057] The following example will provide a more detailed explanation of the above technical solution: Suppose an edge device is a smart surveillance camera, which contains a CPU and an NPU as heterogeneous computing units. This camera needs to run two AI models simultaneously: Model A (object detection model) and Model B (face recognition model). The input of Model B depends on the output of Model A; that is, Model A first detects a face region in the image, and then Model B performs face recognition on that region. In traditional static scheduling schemes, fixed computing units might be pre-allocated to Model A and Model B, for example, Model A would always run on the NPU, and Model B would always run on the CPU, with a fixed execution order. However, in real-world surveillance scenarios, factors such as ambient light, image complexity, and the number of targets dynamically change, causing fluctuations in model inference load, leading to low resource utilization and insufficient inference efficiency.

[0058] The method in this embodiment first dynamically collects hardware resource status data of the intelligent surveillance camera. For example, the system continuously monitors the CPU and NPU computing power utilization, memory usage, and real-time power consumption at a fixed frequency. When the number of targets in the monitored screen suddenly increases, the NPU computing power utilization may rise rapidly, and memory usage will also increase accordingly. This real-time data is captured by the system to provide a basis for subsequent decision-making.

[0059] Simultaneously, the system acquires static metadata and dynamic inference performance data for Model A and Model B. Before deployment, Model A's computational graph structure was analyzed, revealing multiple convolutional and pooling layers, primarily using CNN-related operators, with an input tensor dimension of 1080p video frames. Model B's computational graph structure was analyzed, revealing multiple fully connected layers and activation functions, primarily using DNN-related operators, with an input tensor dimension of face region images. In actual operation, the system, through a one-time benchmark test or historical data, recorded that Model A, when executed on the NPU, had an average latency of 50ms, a throughput of 20 frames / second, and an NPU resource utilization rate of 80%; when executed on the CPU, the average latency was 200ms, the throughput was 5 frames / second, and the CPU resource utilization rate was 60%. Similarly, performance data for Model B on both the NPU and CPU was also acquired.

[0060] Based on the real-time collected hardware resource status data, model static metadata, and dynamic inference performance data, the system dynamically allocates computing resources to Model A and Model B, and determines their inference priorities and execution parallelism strategies, thereby generating a resource scheduling strategy. For example, when the system detects that the NPU's current computing power utilization is low while the CPU's computing power utilization is high, the system will prioritize allocating Model A (computationally intensive) to the NPU. Since Model B depends on the output of Model A, the inference priority of Model A is set higher than that of Model B. The execution parallelism strategy is determined to prevent Model A and Model B from executing in parallel due to data dependencies; however, if Model A contains parallelizable operators, it is allowed to compute in parallel on multiple cores of the NPU. Thus, the system generates an optimized scheduling strategy based on the current load and model dependencies.

[0061] Finally, based on the resource scheduling strategy and model dependencies, the system constructs and executes a system-level inference task pipeline to complete collaborative inference. Specifically, the system determines the execution order based on the dependency between model A and model B (the output of model A is the input of model B): execute model A first, then model B. Subsequently, a directed acyclic computation graph is constructed, where model A and model B are nodes, and edges pointing from model A to model B represent data flows. During graph construction, the system recognizes that the output operators of model A (e.g., the output layer after feature extraction) and the input operators of model B (e.g., the preprocessing layer) are compatible in data format and computation type. Therefore, the system merges these two adjacent and compatible operators to form a composite operator. This composite operator is executed continuously on the NPU, avoiding the intermediate data transfer overhead of writing model A's output data to memory and then having model B read it from memory. This merged system-level inference task pipeline is then submitted to the NPU and CPU for execution, thereby achieving efficient collaborative inference between model A and model B. In this way, even when the monitoring scenario is dynamically changing, the system can adjust resource allocation and execution strategies according to the real-time situation, ensuring the timeliness and efficiency of inference tasks.

[0062] Based on the example of the intelligent surveillance camera described above, the technical solution of this embodiment demonstrates a significant improvement over existing technologies. In traditional edge-side multi-model inference schemes, resource allocation is often static or semi-static. For example, Model A and Model B may be pre-assigned to specific computing units, and their execution order and parallel mode are difficult to change after deployment. When the monitoring scenario changes, such as a surge in the number of targets in the frame leading to increased NPU load, the static allocation scheme cannot be adjusted in time, which may result in a significant increase in the inference latency of Model A and Model B, or even inference task backlog. Under low load conditions, the reserved computing resources may be idle, resulting in resource waste.

[0063] In contrast, this embodiment achieves real-time perception of the operating environment of the edge device and the actual performance of the model by dynamically collecting hardware resource status data and acquiring dynamic inference performance data of the AI ​​model. In the example above, the system can monitor the computing power utilization and memory usage of the NPU and CPU in real time, and combine this with the actual performance data of model A and model B on different computing units. This real-time perception capability allows the system to overcome the limitations of static allocation and provides a data foundation for subsequent dynamic decision-making.

[0064] Furthermore, this embodiment dynamically allocates computing resources to each AI model based on this multi-source data and determines inference priority and parallel execution strategies. In the example, the system can intelligently prioritize allocating model A to the NPU with lower load and better performance based on the real-time load of the NPU and CPU, rather than blindly fixing it. Simultaneously, considering the dependency of model B on model A, the system can reasonably set the inference priority of model A higher than that of model B and plan their execution order, avoiding performance bottlenecks caused by improper handling of dependencies. This dependency-aware dynamic scheduling strategy effectively solves the problem of lack of global collaborative optimization in traditional scheduling strategies, making resource allocation and task execution more flexible and efficient.

[0065] Furthermore, this embodiment further optimizes the efficiency of multi-model collaborative inference by constructing a system-level inference task pipeline and performing operator fusion. In the example, the system not only constructs a directed acyclic computation graph based on model dependencies, but more importantly, it identifies and fuses the output operator of model A with the input operator of model B. In traditional solutions, the output data of model A typically needs to be completely written to memory, and then model B reads it from memory, which involves additional data movement and memory access overhead. Through operator fusion, the composite operator can be executed continuously within the NPU, significantly reducing the number of intermediate data movements between memory and different heterogeneous computing units, thereby reducing inference latency and improving the efficiency of the overall data flow. This fine-grained optimization is difficult to achieve with traditional solutions, and it effectively solves the technical problem of difficulty in achieving cross-model operator fusion and data flow optimization in existing technologies.

[0066] In summary, the technical solution of this embodiment constructs a closed-loop collaborative reasoning framework through dynamic perception, dependency-aware scheduling, and system-level pipeline optimization. This framework can effectively address the challenges of limited resources and dynamically changing loads on edge devices, significantly improving resource utilization and inference efficiency for collaborative inference of multiple AI models, thereby solving the technical problems of low resource utilization and insufficient inference efficiency in existing technologies.

[0067] In other embodiments, this application proposes an edge-side multi-model collaborative inference method applied to an edge device containing multiple heterogeneous computing units, on which multiple AI models are deployed. This method dynamically collects hardware resource status data from the edge device to obtain static metadata and dynamic inference performance data for each AI model. Based on this data, it dynamically allocates computing resources from heterogeneous computing units to each AI model, determining the inference priority and execution parallelism strategy for each AI model to generate a resource scheduling strategy. The execution parallelism strategy indicates whether models without dependencies are executed in parallel, and whether a single model is computed in parallel on multiple computing units. The inference priority and execution parallelism strategy must be decided in conjunction with the model dependencies between the AI ​​models. Subsequently, based on the resource scheduling strategy and model dependencies, a system-level inference task pipeline is constructed and executed to complete collaborative inference. However, in actual execution, due to the complexity of the edge device environment, the dynamic nature of model behavior, and the volatility of hardware status, the preset resource scheduling strategy and inference task pipeline may not fully adapt to real-time changes, resulting in suboptimal inference efficiency or resource utilization.

[0068] In response, this embodiment further proposes that after constructing and executing a system-level inference task pipeline to complete collaborative inference based on resource scheduling strategies and model dependencies, the method also includes: collecting performance indicators during the execution of the system-level inference task pipeline, and feeding back the performance indicators to update dynamic inference performance data, so as to optimize the resource scheduling strategy and inference task pipeline in the next round.

[0069] The collection of performance metrics during the execution of the system-level inference task pipeline refers to acquiring various quantitative data of the system-level inference task pipeline during actual operation to evaluate its operational efficiency and resource consumption. These performance metrics may include, but are not limited to: inference latency, i.e., the time required from task input to result output; throughput, i.e., the number of inference tasks completed or the amount of data processed per unit time; and the actual resource utilization rate of each heterogeneous computing unit, such as computing power utilization rate and memory utilization rate. These metrics can be achieved by setting timestamps and counters at key nodes of the pipeline, or by real-time monitoring and recording through API interfaces provided by the operating system and hardware.

[0070] Using performance metrics as feedback to update dynamic inference performance data refers to using actually collected performance metrics to correct or supplement previously acquired dynamic inference performance data used for decision-making. For example, the latest collected actual latency, throughput, or resource utilization data can be directly replaced or weighted and averaged with historical data to update the dynamic inference performance data of the corresponding AI model on a specific heterogeneous computing unit. Furthermore, this feedback data can also be used as training samples to calibrate or fine-tune the internal prediction model used to predict model performance online, thereby improving prediction accuracy.

[0071] Optimizing the resource scheduling strategy and inference task pipeline for the next round refers to adjusting and improving subsequent resource allocation, inference priority, execution parallelization strategy, and pipeline construction based on updated dynamic inference performance data. For example, the system can rerun the resource scheduling algorithm, which will re-evaluate the performance of each AI model on different heterogeneous computing units based on the latest performance data, thereby generating new and better resource allocation schemes, inference priorities, and execution parallelization strategies. If the feedback performance indicators show that the current pipeline has bottlenecks or is inefficient, the system can also re-evaluate the dependencies between models, the possibility of operator fusion, and parallel execution strategies based on updated data, thereby reconstructing or adjusting the system-level inference task pipeline.

[0072] This application's solution collects performance metrics during the execution of a system-level inference task pipeline after its completion and feeds this actual operational data back to the system to update the dynamic inference performance data of the AI ​​model. This feedback mechanism enables the system to continuously learn and adapt, taking into account potential deviations or unforeseen situations that may occur during actual operation. Based on the updated dynamic inference performance data, the system can more accurately evaluate the real-world performance of each AI model on different heterogeneous computing units, thereby making more informed decisions in the next round of resource scheduling, such as adjusting the allocation of computing resources, optimizing inference priorities and execution parallel strategies, and even reconstructing the inference task pipeline. This closed-loop adaptive optimization process allows edge devices to overcome the limitations of the initial scheduling strategy, continuously adjust and improve their multi-model collaborative inference capabilities, and ensure high efficiency and robustness in dynamically changing edge environments.

[0073] The following example illustrates this. Suppose an edge device needs to run an AI model A for image classification and an AI model B for object detection simultaneously. Initially, the system allocates model A to the NPU and model B to the GPU based on preset dynamic inference performance data, and constructs the corresponding inference task pipeline. During pipeline execution, the system collects performance metrics in real time, such as the actual inference latency of model A on the NPU, the NPU's computing power utilization, and the actual throughput and GPU memory usage of model B on the GPU. If the system detects that the actual inference latency of model A on the NPU is consistently higher than expected, or the NPU's computing power utilization far exceeds a preset threshold, these performance metrics are fed back to the system. The system uses this feedback data to update the dynamic inference performance data of model A on the NPU, for example, by lowering the NPU's assessment of model A's actual processing capacity. In the next round of resource scheduling, the scheduler will reassess the resource requirements of model A and model B and the availability of heterogeneous computing units based on this updated data. For example, the scheduler might decide to migrate some tasks of model A to the CPU for execution, or adjust the inference priority of model A to reduce the burden on the NPU, thereby optimizing the overall resource allocation and inference efficiency.

[0074] Through the above technical solutions, the edge-side multi-model collaborative inference method can achieve adaptive optimization. The system no longer relies solely on preset or static performance data for scheduling, but can dynamically adjust and learn based on actual operating conditions. This enables the system to better cope with fluctuations in edge device hardware status, dynamic changes in AI model behavior, and uncertainties in the external environment, thereby continuously improving the overall performance of multi-model collaborative inference, including reducing inference latency, increasing throughput, and optimizing resource utilization, ensuring that edge devices provide stable and efficient AI services in complex application scenarios.

[0075] In edge-side multi-model collaborative inference, dynamically collecting hardware resource status data from edge devices is fundamental for effective resource scheduling. However, using a fixed collection cycle presents a dilemma: a collection cycle that is too short increases system overhead, consuming valuable computing resources and impacting inference performance; while a collection cycle that is too long may lead to untimely responses to changes in hardware resource status, failing to accurately reflect the current system load and the actual needs of AI model inference tasks, thus affecting the accuracy and efficiency of resource allocation. This fixed-cycle collection method is ill-suited to the dynamically changing operating environment of edge devices and the complexity of multi-model inference tasks.

[0076] In response, this application further proposes a step for dynamically collecting hardware resource status data of the terminal device, including: collecting hardware resource status data with a dynamically adjustable collection cycle; wherein the collection cycle is adjusted based on the current system load or the urgency of the AI ​​model's inference task; when the system load is higher than a preset threshold or there are tasks with an inference task urgency greater than a preset value, the collection cycle is shortened to increase the monitoring frequency; when the system load is lower than a preset threshold and there are no tasks with an inference task urgency greater than a preset value, the collection cycle is extended to reduce system overhead.

[0077] The "dynamically adjustable acquisition cycle" refers to the time interval used to periodically acquire hardware resource status data from the edge device. This time interval is not fixed but can be changed in real-time or near real-time based on system operating status or specific conditions. For example, software modules can modify their internal timers or scheduling parameters according to preset logic or algorithms during runtime, thereby changing the trigger frequency of data acquisition tasks. Alternatively, the operating system-level scheduler can provide API interfaces, allowing upper-layer applications to dynamically adjust the execution frequency of specific tasks, such as indirectly affecting the acquisition cycle by modifying the interval of timer interrupts or task priorities. "The adjustment of the acquisition cycle is based on the current system load or the urgency of the AI ​​model's inference tasks." These bases are the core inputs for deciding on acquisition cycle adjustments, ensuring the rationality and effectiveness of the adjustments. The current system load refers to the overall usage of computing, memory, network, and other resources of the edge device, reflecting the device's workload. This can be comprehensively assessed by monitoring indicators such as CPU utilization, memory usage, and I / O bandwidth. The urgency of an AI model's inference task refers to the requirements of a specific AI model's inference task regarding latency, real-time performance, etc. For example, perception model inference tasks in autonomous driving typically have a high urgency, while backend data analysis models may have a lower urgency. This can be defined through task metadata, user configuration, or preset business rules. When the system load exceeds a preset threshold or there are tasks with an inference task urgency greater than the preset value, the data collection cycle is shortened to increase the monitoring frequency. This is beneficial because when system resources are strained or critical tasks require high real-time performance, more frequent data collection allows for more timely and accurate acquisition of the latest information on hardware resource status. For example, when CPU utilization exceeds 80% for N consecutive seconds (a preset threshold) or an AI model inference task marked as "high priority" enters the execution queue, the data collection cycle is shortened from the default 500 milliseconds to 20 milliseconds. Alternatively, the system can maintain a dynamic threshold; when the short-term average of the system load exceeds this threshold, the data collection cycle is shortened. When the system load is below a preset threshold and there are no inference tasks with an urgency level greater than a preset value, the data acquisition cycle is extended to reduce system overhead. This reduces the frequency of data acquisition when system resources are idle or tasks are not urgent, thereby reducing system resource consumption, saving power, and freeing up resources for other computing tasks. For example, if CPU utilization is below 20% (the preset threshold) for M consecutive seconds and there are no high-urgency AI model inference tasks running, the acquisition cycle is extended from 20 milliseconds to 500 milliseconds. Alternatively, the system can employ an adaptive algorithm to gradually extend the acquisition cycle based on historical load trends and task urgency predictions, as the system is expected to enter an idle state, thus smoothly reducing system overhead.

[0078] This embodiment resolves the conflict between efficiency and accuracy caused by a fixed acquisition cycle by introducing a dynamically adjusted hardware resource status data acquisition mechanism. When performing multi-model collaborative inference on the edge device, the system no longer acquires hardware resource status data at a single, fixed frequency. Instead, it intelligently adjusts the acquisition frequency based on the actual system operation. Specifically, when the overall system load on the edge device is high—for example, when the computing power utilization, memory usage, and power consumption of each heterogeneous computing unit remain high—or when there are inference tasks for AI models with extremely high real-time requirements, the system proactively shortens the hardware resource status data acquisition cycle. This means that at critical moments, the system can perceive changes in hardware resources at a higher frequency and with finer granularity, providing the latest and most accurate basis for subsequent dynamic allocation of AI model computing resources, determination of inference priorities, and generation of parallel execution strategies. This allows resource scheduling strategies to respond more quickly to fluctuations in system status, avoiding improper resource allocation or low task execution efficiency due to information lag. Conversely, when the system load is low and there are no urgent AI model inference tasks to process, the system extends the hardware resource status data acquisition cycle. This strategic reduction in data acquisition frequency effectively minimizes system overhead associated with data acquisition itself, including CPU cycles, memory usage, and power consumption. This frees up more computing resources for AI model inference and reduces overall device energy consumption. Through this dynamic and adaptive acquisition mechanism, the proposed solution ensures that while maintaining accurate resource scheduling and response speed, it maximizes the utilization efficiency of system resources and minimizes overall operational overhead, providing more flexible and efficient basic data support for edge-side multi-model collaborative inference.

[0079] The following is a concrete example. An edge device can deploy a resource monitoring agent module, responsible for collecting hardware resource status data and dynamically adjusting the collection cycle. This agent module can continuously monitor the computing power utilization, memory usage, and power consumption of heterogeneous computing units such as CPU, GPU, and NPU. For example, the agent module can be set to a default collection cycle, such as collecting data every 5 seconds. When the system detects current system load, such as a CPU average utilization rate consistently exceeding a preset threshold of 80% over the past 30 seconds, or when inference requests from high-urgency AI models (such as object detection models) in autonomous driving applications enter the queue, the agent module will immediately shorten the collection cycle to 50 milliseconds. This means that during periods of system busyness or critical task execution, resource status data will be updated more frequently, ensuring that the resource scheduler can obtain the latest hardware information in a timely manner. Conversely, when system load remains low, such as a CPU average utilization rate below a preset threshold of 20% over the past 60 seconds, and there are currently no high-urgency AI model inference tasks being executed, the agent module will extend the collection cycle to 500 milliseconds. This extended data collection cycle effectively reduces the computational and memory overhead of the agent module itself, lowering the overall system power consumption and freeing up more resources for other non-urgent AI model inference tasks or background services. The agent module can also be configured via an interface, allowing system administrators or upper-layer applications to adjust preset system load thresholds and inference task urgency criteria to suit different application scenarios and performance requirements.

[0080] Through the above technical solution, the edge device can intelligently adjust the frequency of hardware resource status data collection based on actual operating conditions. When the system load is high or there are urgent inference tasks, the collection cycle is shortened to ensure the resource scheduler can obtain accurate hardware status information in a timely manner. This enables more precise and real-time allocation of computing resources and task scheduling, avoiding resource waste or task delays caused by information lag. Conversely, when the system load is low and there are no urgent tasks, the collection cycle is extended, effectively reducing the system overhead and power consumption caused by data collection itself, freeing up valuable computing resources, and improving the overall energy efficiency of the edge device. This adaptive collection mechanism allows the edge multi-model collaborative inference system to ensure high performance and high responsiveness while also optimizing the utilization of system resources, significantly improving the operating efficiency and stability of the edge device in complex dynamic environments.

[0081] In some of the embodiments described above in this application, a multi-model collaborative inference method for edge devices is proposed. This method dynamically collects hardware resource status data, obtains static metadata and dynamic inference performance data of AI models, and dynamically allocates computing resources, determines inference priorities and execution parallel strategies based on this data, thereby constructing and executing a system-level inference task pipeline. However, on edge devices containing various heterogeneous computing units (such as CPUs, GPUs, and NPUs), how to uniformly and efficiently manage and allocate these different types of computing resources to ensure the effectiveness and accuracy of dynamic resource scheduling remains a problem that needs to be solved.

[0082] In response, this application further proposes the aforementioned edge-side multi-model collaborative reasoning method, please refer to... Figure 2 The heterogeneous computing units include at least two of the following: CPU, GPU, and NPU. The steps for dynamically allocating computing resources to each AI model based on hardware resource status data, static metadata, and dynamic inference performance data include: Step S500: Establish and maintain a global resource pool to perform unified abstraction, quantification and state management of at least two types of computing resources, including CPU, GPU and NPU. Step S600: Based on hardware resource status data, static metadata and dynamic inference performance data, and combined with the real-time available resources of the global resource pool, computing resources are dynamically allocated to each AI model.

[0083] Specifically, heterogeneous computing units refer to computing units that integrate different types of processor cores within the same system, such as central processing units (CPUs), graphics processing units (GPUs), and neural network processing units (NPUs). CPUs excel at general-purpose computing and control logic, GPUs excel at parallel floating-point operations, and NPUs are specifically optimized for neural network computing. Edge devices typically integrate these different types of computing units to meet the diverse inference needs of AI models. For example, some models may be better suited to run on an NPU for high energy efficiency, while others may perform better on a GPU or handle control flow on a CPU. Common edge heterogeneous computing platforms include Qualcomm Snapdragon series chips, Huawei Ascend series chips, and MediaTek Dimensity series chips. These platforms support the collaborative work of multiple computing units through hardware design and software stacks.

[0084] A global resource pool is a logical concept used to centrally manage the computing resources of all heterogeneous computing units on edge devices. Its core lies in the "unified abstraction," "quantization," and "state management" of different types of computing resources. Unified abstraction means shielding the underlying differences between different hardware such as CPUs, GPUs, and NPUs, providing a unified resource view and interface to the upper layers. For example, the computing power of different computing units can be uniformly abstracted as "computing units" or "computing power points," and memory can be abstracted as "memory blocks." Quantization refers to representing the abstracted resources numerically to facilitate comparison, allocation, and scheduling. For example, the computing power of a CPU can be quantified as MIPS or GFLOPS, the computing power of GPUs and NPUs can be quantified as TOPS or GFLOPS, and memory can be quantified as MB or GB. State management refers to the real-time tracking and updating of information such as the available resources, utilization rate, and health status of each computing unit. This includes recording the currently allocated resources, remaining available resources, and running status of each computing unit. This can be achieved through an operating system-level resource manager, a dedicated runtime environment (such as the unified API provided by TensorFlow Lite or PyTorchMobile), or a custom scheduler module.

[0085] In scenarios where edge devices contain at least two types of heterogeneous computing units such as CPUs, GPUs, and NPUs, this application achieves unified management of these heterogeneous computing resources by establishing and maintaining a global resource pool. Specifically, the global resource pool abstracts the underlying hardware characteristics of different types of computing units and quantifies their computing power, memory, and other resources, while simultaneously tracking and managing the usage status and availability of these resources in real time. This unified abstraction, quantification, and status management mechanism enables the upper-layer scheduler to obtain a clear and comprehensive view of resources, thereby overcoming the drawbacks of traditionally fragmented and difficult-to-coordinate heterogeneous resource management. Based on this, when it is necessary to dynamically allocate computing resources to various AI models, the system no longer relies solely on independent hardware resource status data, static metadata, and dynamic inference performance data, but further incorporates the real-time available resource quantity provided by the global resource pool. This means that when making allocation decisions, the scheduler not only considers the characteristics and performance requirements of the AI ​​model itself, but also fully considers the actual load and remaining resource status of all heterogeneous computing units on the entire device. For example, a computationally intensive and latency-sensitive AI model will be prioritized for execution on the NPU if the global resource pool indicates that the NPU is currently under low load and has sufficient resources. Conversely, a model with high general-purpose computing requirements and limited NPU or GPU resources may be allocated to the CPU. This dynamic allocation method, combined with the global resource pool, ensures the global optimality of resource allocation decisions, avoiding local resource overload or idleness, thereby significantly improving the overall utilization of heterogeneous computing units and the efficiency of AI model collaborative inference. By feeding back performance metrics to update dynamic inference performance data and optimize the resource scheduling strategy and inference task pipeline for the next round, the real-time available resource information of the global resource pool can also be continuously optimized. This allows the resource allocation strategy to adaptively adjust with changes in system operating status and model performance, forming a closed-loop optimization process.

[0086] As a specific implementation, assume an edge device integrates a quad-core CPU, a GPU, and an NPU. The system first establishes and maintains a global resource pool. This pool provides unified abstraction, quantification, and real-time state management of the CPU's computing power (e.g., in MIPS), memory (in MB), GPU's computing power (e.g., in GFLOPS), video memory (in MB), and NPU's computing power (e.g., in TOPS) and dedicated memory (in MB). For example, the global resource pool might record that the CPU's current available computing power is 80%, and available memory is 1.5GB; the GPU's available computing power is 40%, and available video memory is 500MB; and the NPU's available computing power is 70%, and available dedicated memory is 300MB. At this point, two AI models need to perform inference: the first is an image recognition model, whose static metadata shows it contains a large number of convolution operators, and dynamic inference performance data shows it runs most efficiently and consumes the least power on the NPU; the second is a natural language processing model, whose static metadata shows it contains a large number of matrix multiplications and fully connected operators, and dynamic inference performance data shows it runs most efficiently on the GPU, followed by the CPU, with the NPU being unsuitable. When dynamically allocating computing resources to these two AI models, the system makes decisions based on previously collected hardware resource status data, the model's static metadata and dynamic inference performance data, combined with the real-time available resources in the global resource pool. Specifically, the scheduler prioritizes allocating the image recognition model to the NPU, because the NPU is best suited for processing convolution operators, and the global resource pool shows that the NPU currently has 70% of its computing power available, which is sufficient to meet the model's needs. Simultaneously, the scheduler checks the available resources of the GPU; if the GPU's available computing power (40%) and video memory (500MB) can meet the needs of the natural language processing model, it is allocated to the GPU. If GPU resources are scarce, the scheduler will consider allocating the natural language processing model to the CPU, based on the CPU's available computing power (80%) and memory (1.5GB). After allocation, the global resource pool will immediately update the available resources of each heterogeneous computing unit for subsequent resource allocation decisions.

[0087] Through the above technical solution, this application effectively solves the problem of fragmented heterogeneous resource management and difficulty in unified scheduling in complex environments where edge devices include various heterogeneous computing units such as CPUs, GPUs, and NPUs. By establishing and maintaining a global resource pool, it achieves unified abstraction, quantification, and state management of different types of computing resources, providing a comprehensive and real-time resource view for the upper-layer scheduler. This allows for more accurate and efficient decision-making when dynamically allocating computing resources to AI models, taking into full account the actual load and available resources of each heterogeneous computing unit. For example, it avoids allocating models suitable for NPU operation to overloaded CPUs or underutilizing idle GPU resources. This refined resource management and allocation mechanism significantly improves the overall utilization rate of heterogeneous computing units on edge devices, reduces the average latency of AI model collaborative inference, increases system throughput, and helps optimize power consumption, thus providing a more stable and efficient operating environment for edge-side multi-model collaborative inference.

[0088] When performing multi-model collaborative inference on edge devices, although computing resources can be dynamically allocated and inference priority and parallel execution strategies can be determined based on model dependencies, the hardware resource status of edge devices and the inference performance data of AI models are constantly changing. If decisions are made based solely on current or instantaneous data, the resource scheduling strategy may not be forward-looking enough and may not be able to effectively cope with future fluctuations in computing power demand, thereby affecting the overall efficiency and response speed of the inference task.

[0089] For this, please refer to Figure 3 This application further proposes steps for determining the inference priority of each model and the parallel execution strategy, which specifically include: Step S700: Based on the historical sequence of dynamic inference performance data, an adaptive load balancing algorithm is used to predict the changes in computing power demand of each AI model in the next time window. Step S800: Combine the predicted changes in computing power demand, model dependencies, and preset task urgency indicators to calculate and dynamically adjust the inference priority of each model.

[0090] The historical sequence of dynamic inference performance data refers to a continuous record of performance indicators such as actual latency, throughput, and resource utilization on heterogeneous computing units for each AI model collected by the system over a past period. This historical data provides a foundation for analyzing model behavior patterns and predicting future trends. For example, the system can maintain a sliding window to store performance data from the most recent N time steps, or persist this data to local storage for long-term analysis. An adaptive load balancing algorithm is an algorithm that dynamically adjusts its behavior based on the system's real-time state and historical data. This algorithm aims to optimize resource allocation and task scheduling to improve overall system performance. For example, this algorithm can be a machine learning-based model, such as a recurrent neural network (RNN) or a long short-term memory network (LSTM), which learns time-series patterns in historical performance data to predict future computing power needs; or it can be a reinforcement learning-based agent that learns the optimal scheduling strategy by interacting with the environment and receiving reward signals. Predicting the changes in computing power needs of each AI model in the next time window refers to using the aforementioned adaptive load balancing algorithm, combined with historical performance data, to estimate the amount of computing resources required by each AI model in a specific future time period. For example, an algorithm can predict the number of floating-point operations (GFLOPS) or CPU core utilization required by an AI model in the next 30 seconds. This prediction can be periodic, such as making predictions at regular intervals, or event-driven, such as being triggered when a significant change in system load is detected. A preset task urgency metric is a quantified value pre-set for the inference task of each AI model, representing the urgency or importance of the task. For example, for perception models in autonomous driving scenarios, the task urgency metric might be set to the highest level because it directly relates to driving safety; while for some background data analysis models, the task urgency metric might be lower. This metric can be a discrete level value (e.g., 1 to 5) or a continuous value with a deadline. Calculating and dynamically adjusting the inference priority of each model refers to comprehensively considering changes in prediction computing power requirements, model dependencies, and preset task urgency metrics, using a decision logic or optimization algorithm to determine the execution order of each AI model at the current moment. For example, the system can use a multi-objective optimization function to assign higher priority to models with high prediction computing power requirements, strong dependencies, and high task urgency. Dynamic adjustment means that this priority is not fixed, but is updated in real time based on system status and prediction results to ensure that critical tasks can obtain the necessary computing resources in a timely manner.

[0091] This application's solution transforms traditional passive resource scheduling into proactive, predictive scheduling by introducing historical sequence analysis of dynamic inference performance data and employing an adaptive load balancing algorithm to predict changes in computing power demand for each AI model in the next time window. Building upon the aforementioned dynamic acquisition of hardware resource status data from edge devices, obtaining static metadata and dynamic inference performance data of AI models, dynamically allocating computing resources to AI models based on this data, and establishing and maintaining a global resource pool, this solution further combines predicted changes in computing power demand, model dependencies, and preset task urgency indicators to more accurately calculate and dynamically adjust the inference priority of each model. This forward-looking priority adjustment mechanism enables the system to anticipate and respond to potential resource bottlenecks or task peaks in the future, thereby making better decisions in resource allocation and task scheduling, effectively improving the overall efficiency and stability of edge-side multi-model collaborative inference.

[0092] The following is a concrete example. Assume an autonomous driving system running on an edge device, comprising multiple AI models such as an object detection model, a path planning model, and a driver state monitoring model. The system continuously collects dynamic inference performance data from these models and stores it as historical sequences. For example, it records the actual latency and computational resource utilization of each model on the NPU once per second. An adaptive load balancing algorithm deployed on the device, such as a lightweight LSTM model, periodically (e.g., every 5 seconds) analyzes these historical sequences to predict the changes in computational resource requirements of the object detection model, path planning model, and driver state monitoring model over the next 10 seconds. Simultaneously, the system presets task urgency indicators: highest for the object detection model (e.g., 5), followed by the path planning model (e.g., 4), and lowest for the driver state monitoring model (e.g., 3). When the LSTM model predicts a significant increase in computational resource requirements for the object detection model in the next time window, the scheduler, considering its high task urgency and the path planning model's dependency on it, immediately increases the inference priority of the object detection model. Even if the current system load is not high, this predictive adjustment can ensure that sufficient computing resources are reserved or prioritized for critical models before the actual demand peaks, thereby avoiding inference delays caused by resource contention.

[0093] Through the aforementioned technical solution, edge devices can shift from passively responding to resource changes to proactively predicting and planning. This forward-looking priority adjustment mechanism enables the system to more effectively manage limited heterogeneous computing resources, significantly reducing latency for critical inference tasks and improving overall system throughput and response speed. Especially in resource-constrained edge environments with dynamically changing task loads, this solution ensures the timely completion of high-priority and high-urgency tasks, thereby enhancing the robustness and reliability of multi-model collaborative inference.

[0094] Furthermore, this application proposes to compare the collected actual latency and throughput with the expected performance under the current resource scheduling strategy to generate prediction error; based on the prediction error, an online learning mechanism is used to adjust the internal weight parameters of the adaptive load balancing algorithm used to predict changes in computing power demand.

[0095] Specifically, the actual latency and throughput collected are compared with the expected performance under the current resource scheduling strategy to generate a prediction error, aiming to quantify the prediction accuracy of the resource scheduling strategy. Actual latency and throughput directly reflect the actual performance of the system, while expected performance is the ideal performance estimated based on the current resource scheduling strategy and prediction model. By comparing the two, a quantified prediction error can be obtained, reflecting the degree of deviation between the prediction model and the actual situation. For example, the prediction error can be generated by calculating the absolute or relative difference between the actual and expected values, or by using statistical indicators such as mean squared error (MSE) or mean absolute error (MAE) to comprehensively evaluate the prediction errors of multiple performance indicators, thus obtaining a more comprehensive error metric. Based on the prediction error, an online learning mechanism is used to adjust the internal weight parameters of the adaptive load balancing algorithm used to predict changes in computing power demand, which is the core adaptive optimization mechanism. The online learning mechanism allows the system to continuously adjust model parameters based on new data and feedback during operation, enabling the adaptive load balancing algorithm for predicting computing power demand to continuously learn and adapt to the dynamically changing operating environment of the edge devices and the inference characteristics of the AI ​​model. By adjusting internal weight parameters, algorithms can better capture patterns in performance changes and improve the accuracy of future computing power demand predictions. For example, gradient descent or its variants (such as stochastic gradient descent (SGD), Adam, etc.) can be used as an online learning mechanism. After each prediction error is generated, the weight parameters affecting the prediction results in the adaptive load balancing algorithm are fine-tuned based on the error, optimizing them in the direction of reducing error. Alternatively, reinforcement learning can be used, treating the prediction error as a reward or penalty signal. Through interaction between the agent and the environment, the algorithm learns and adjusts the strategy (i.e., weight parameters) of the adaptive load balancing algorithm to maximize long-term prediction accuracy.

[0096] The proposed solution generates a quantified prediction error by precisely comparing actual performance metrics collected after the system executes a system-level inference task pipeline with the performance expected by the current resource scheduling strategy. This prediction error not only reflects the effectiveness of the current scheduling strategy but, more importantly, serves as a learning signal, driving an online learning mechanism to dynamically adjust the internal weight parameters of the adaptive load balancing algorithm used to predict changes in computing power demand. This continuous, feedback-based parameter optimization allows the prediction algorithm to learn and adapt to dynamic changes in edge device hardware resource status, AI model inference characteristics, and the external environment. Therefore, this mechanism ensures that the accuracy and robustness of the prediction algorithm improve over time, enabling subsequent resource scheduling strategies to more accurately match actual computing power demands, effectively utilize the computing resources of heterogeneous computing units, and optimize the overall performance of the system-level inference task pipeline.

[0097] The following is a concrete example. Assume multiple AI models are running on the edge device for real-time video streaming, such as object detection, face recognition, and behavior analysis. The system uses an adaptive load balancing algorithm based on historical dynamic inference performance data sequences to predict the computing power requirement of each AI model within the next time window (e.g., 100 milliseconds). After the system generates and executes a resource scheduling strategy based on the predicted computing power requirement, it collects the actual inference latency and throughput. For example, if the expected inference latency for the object detection model is 50 milliseconds, but the actual latency is 55 milliseconds, a positive prediction error is generated. Similarly, if the expected throughput is 20 frames per second, but the actual throughput is 18 frames per second, a negative prediction error is generated. These prediction errors are then fed into an online learning module. This module can use a lightweight gradient descent algorithm to fine-tune the internal weight parameters related to latency and throughput prediction in the adaptive load balancing algorithm based on these error signals. For example, if the algorithm consistently underestimates latency, it will adjust the relevant weights to make them more likely to give higher latency estimates in future predictions. Through this continuous feedback and adjustment, the adaptive load balancing algorithm can continuously optimize its predictive capabilities, enabling it to provide accurate predictions of computing power requirements even when faced with constantly changing video content, model load, or hardware status, thereby ensuring the effectiveness of resource scheduling strategies.

[0098] Through the above technical solution, this application can quantify the accuracy of the current resource scheduling strategy and, based on this prediction error, dynamically adjust the internal weight parameters of the adaptive load balancing algorithm used to predict changes in computing power demand using an online learning mechanism. This adaptive optimization mechanism significantly improves the accuracy and robustness of computing power demand prediction, thereby ensuring that the generation of subsequent resource scheduling strategies is more accurate and efficient. Ultimately, this helps to more effectively utilize the computing resources of heterogeneous computing units, reduce resource waste, and improve the overall performance of the system-level inference task pipeline, such as reducing inference latency, increasing throughput, and enhancing the stability and reliability of the system in complex and variable environments.

[0099] On edge devices, this application proposes a multi-model collaborative inference method. This method dynamically collects hardware resource status data, acquires static metadata and dynamic inference performance data of AI models, and dynamically allocates computing resources from heterogeneous computing units to each AI model based on this information. Simultaneously, it determines inference priorities and execution parallelism strategies to generate a resource scheduling strategy, thereby constructing and executing a system-level inference task pipeline to complete collaborative inference. Furthermore, this method collects performance metrics during pipeline execution and feeds them back to update dynamic inference performance data, optimizing the resource scheduling strategy and inference task pipeline for the next round. However, in actual deployments, heterogeneous computing units on edge devices may malfunction or experience performance degradation for various reasons, potentially interrupting ongoing inference tasks and affecting system stability and reliability.

[0100] To address this, this application further proposes a fault recovery mechanism. This mechanism refers to the system's ability to automatically or semi-automatically take measures when it detects an anomaly or fault, minimizing the impact of the fault on system functionality and performance, and restoring normal operation as much as possible. Its purpose is to improve the system's robustness and availability. This mechanism can be implemented through a pre-defined fault handling strategy library containing predefined recovery actions for different types of faults; alternatively, it can be implemented through a machine learning-based adaptive recovery strategy, where the system dynamically adjusts the recovery strategy based on historical fault data and recovery results.

[0101] To implement the aforementioned fault recovery mechanism, the method of this application includes real-time monitoring of the health status of each heterogeneous computing unit. The purpose of real-time monitoring is to ensure that the system can promptly detect potential hardware problems, providing a prerequisite for fault recovery. This can be achieved by periodically querying the hardware registers, system logs, driver interfaces, or dedicated monitoring agents of the heterogeneous computing units to obtain their operational status information, such as temperature, voltage, error count, and response time. Furthermore, lightweight diagnostic programs or heartbeat mechanisms can be run on the heterogeneous computing units to continuously verify their functionality and responsiveness.

[0102] When a fault or severe performance degradation is detected in a heterogeneous computing unit that hosts the computing resources allocated to execute any AI model, the system triggers fault recovery. The criteria for determining a fault or severe performance degradation can be based on preset thresholds. For example, a heterogeneous computing unit is considered faulty or experiencing performance degradation when its error rate exceeds a certain percentage or when the average latency of its processed tasks consistently exceeds a certain threshold. Alternatively, trend analysis or anomaly detection algorithms can be used to identify abnormal fluctuations or continuous deterioration trends in the performance of heterogeneous computing units.

[0103] Once a fault or performance degradation is detected, the system will, based on the real-time available resources in the global resource pool, migrate the affected model inference tasks to other available heterogeneous computing units for continued execution, or reduce the inference accuracy of the affected model inference tasks to reduce computing power requirements. Task migration aims to provide a primary recovery strategy in the event of a fault, ensuring the continuity of inference tasks. The migration process may include: pausing the task on the affected unit, saving the current task state, finding a heterogeneous computing unit with sufficient available resources in the global resource pool, loading the task state and model onto the new unit, and resuming execution from the point of interruption. Another approach is to employ a hot or cold backup mechanism, preparing copies of the model in advance on other units, and immediately switching to the backup unit if the primary unit fails. Reducing inference accuracy provides another fault recovery strategy, suitable for situations where resources are scarce or tasks cannot be fully migrated, ensuring task completion by sacrificing accuracy. Reducing inference accuracy can be achieved in various ways, such as switching to a model version with a higher degree of quantization, using a pruned model, or reducing the number of model layers. Computing power requirements can also be reduced by dynamically adjusting the model's input resolution, batch size, or skipping certain non-critical post-processing steps.

[0104] The fault recovery mechanism in this application continuously monitors the health status of each heterogeneous computing unit in the edge device to ensure that the system can promptly detect potential hardware problems. Once a fault or severe performance degradation of a heterogeneous computing unit is detected, the system immediately triggers a preset recovery strategy. At this time, the system queries and utilizes the real-time available resource information maintained in the global resource pool to assess the overall resource status of the edge device. Based on the assessment results and the preset strategy, the system can adopt two main recovery paths: If there are other healthy heterogeneous computing units with sufficient computing power in the global resource pool, the system will strip the affected AI model inference task from the faulty unit and migrate it to these available heterogeneous computing units for continued execution, aiming to maximize the original performance and accuracy of the inference task and ensure service continuity. If the available resources in the global resource pool are insufficient to support the complete migration of the task, or in order to respond quickly to the fault, the system can choose to reduce the inference accuracy of the affected model inference task. By reducing the accuracy, the computing power required for the task can be significantly reduced, allowing it to continue running on the currently limited available resources, avoiding complete task interruption and ensuring service availability, although it may be accompanied by some performance or accuracy loss. The combination of these two strategies enables the system to flexibly select the optimal recovery plan based on actual resource conditions and business needs when facing hardware failures. This significantly improves the robustness and reliability of the edge-side multi-model collaborative inference system and avoids the risk of the entire inference pipeline being interrupted due to a single hardware failure.

[0105] As a specific implementation, the monitoring agent deployed on the edge device can report the health status data of each heterogeneous computing unit to the central scheduling module every 500 milliseconds, including the temperature, error rate, and task queue length of the CPU, GPU, and NPU. The central scheduling module maintains a list of health thresholds. For example, when the error rate of an NPU exceeds 1% three times consecutively or its task queue length exceeds 100ms for an extended period, the NPU is considered to be in a state of severe performance degradation. Once NPU performance degradation is detected, the central scheduling module immediately queries the global resource pool to obtain the real-time available computing power of the CPU and GPU. Assuming the currently affected AI model is an image recognition model with an original inference accuracy of FP16, if the global resource pool shows that the CPU or GPU has sufficient idle resources to support the image recognition model (e.g., idle computing power exceeds 120% of the computing power required by the model), the central scheduling module will migrate the inference task of the image recognition model from the NPU to an available GPU for continued execution. The migration process includes copying the model weights and current batch data from the NPU's memory to the GPU's memory and restarting the inference process on the GPU. If the global resource pool shows that the available CPU and GPU resources are insufficient to support an FP16 precision image recognition model, or if the migration takes too long, the central scheduling module will instruct the image recognition model to switch to the INT8 quantized version for inference. The INT8 model requires significantly less computing power than the FP16 model, thus allowing inference tasks to continue under the current limited resources, ensuring uninterrupted image recognition service, although recognition accuracy may decrease slightly.

[0106] By introducing a fault recovery mechanism, this application effectively addresses the potential for faults or performance degradation in heterogeneous computing units on edge devices during actual operation. This mechanism, through real-time monitoring of hardware health status and combined with the real-time availability of resources in the global resource pool, can promptly detect and respond to hardware anomalies. In the event of a fault, the system can flexibly choose to migrate affected model inference tasks to other available computing units or reduce computational power requirements by lowering inference accuracy, thereby ensuring the continuity of inference tasks and the availability of the system. This significantly improves the robustness and reliability of the edge-side multi-model collaborative inference system, avoids the risk of the entire inference pipeline being interrupted due to a single hardware failure, and guarantees the stable operation of edge-side AI applications.

[0107] In edge-side multi-model collaborative inference, while constructing a system-level inference task pipeline can involve building a directed acyclic computation graph and fusing operators based on model dependencies, improper operator fusion—such as fusing incompatible or mismatched operators—can not only fail to effectively reduce intermediate data transmission but may also introduce additional processing overhead or even lead to inference errors. This is particularly true when moving data between heterogeneous computing units, where improper fusion can significantly reduce overall inference efficiency. Therefore, ensuring the effectiveness and profitability of operator fusion is a key challenge in improving the performance of edge-side multi-model collaborative inference.

[0108] In response, this application further proposes a step for operator fusion of adjacent and operator-compatible model nodes in a directed acyclic computation graph. Specifically, it includes: determining whether the output operator of the upstream model node in the directed acyclic computation graph matches the input operator of the downstream model node in the directed acyclic computation graph in terms of data precision, data dimension, and computation type; if the matching degree exceeds a preset fusion threshold, the output operator and the input operator are fused into a composite operator, which is executed continuously within a computation unit to reduce the number of times intermediate data is moved between memory and different heterogeneous computation units.

[0109] The process involves determining whether the output operators of upstream model nodes in a directed acyclic computation graph (DAG) match the input operators of downstream model nodes in terms of data precision, data dimension, and computation type. This aims to evaluate the feasibility and potential benefits of fusing two adjacent operators. By checking data precision, data dimension, and computation type, it ensures that the fused composite operator can correctly process the data flow, avoiding errors caused by inconsistent data formats or incompatible computational logic. This can be achieved by comparing the data type of the upstream operator's output tensor (e.g., float32, int8) with the required data type of the downstream operator's input tensor, or by comparing the shape of the upstream operator's output tensor (e.g., [batch_size, channels, height, width]) with the expected shape of the downstream operator's input tensor. Furthermore, the compatibility of computation types can be determined by analyzing whether the output operations of the upstream operator and the input operations of the downstream operator can be logically seamlessly connected. If the matching degree exceeds a preset fusion threshold, the output operator and the input operator are fused into a composite operator. This composite operator is executed continuously within a single computing unit to reduce the number of times intermediate data is moved between memory and different heterogeneous computing units. This step is the core operation of operator fusion. When two operators are determined to have sufficient compatibility, they are logically merged into a composite operator, which is then forced to execute within a single computing unit. Its main purpose is to eliminate or significantly reduce the storage and transmission overhead of intermediate results. In heterogeneous computing environments, data transmission between different computing units is often one of the performance bottlenecks. Continuous execution within a single unit after fusion can avoid this expensive transmission. The fusion operation can be implemented by modifying the structure of the computation graph, for example, by replacing two independent operator nodes with a new composite operator node. The preset fusion threshold can be an empirical value or dynamically adjusted through performance analysis based on actual test results. To ensure that the composite operator executes continuously within a single computing unit, the scheduler can force all sub-operations of the composite operator to be assigned to the same heterogeneous computing unit.

[0110] This application's solution intelligently identifies adjacent and operator-compatible model nodes in a directed acyclic computation graph and determines the matching degree between their output and input operators, ensuring that only truly compatible operators that can generate benefits are fused. This strict matching mechanism based on data precision, data dimension, and computation type avoids errors or additional overhead that may result from improper fusion. Once the matching degree reaches a preset threshold, the system fuses these operators into a composite operator and, crucially, forces this composite operator to execute continuously within a single computational unit. This strategy directly addresses the performance bottleneck caused by frequent data movement between memory and different heterogeneous computational units in edge-side heterogeneous computing environments. By localizing related computations to a single computational unit, data does not need to be frequently written to main memory or undergo expensive transfers between different types of computational units, thus significantly reducing memory bandwidth pressure and cross-unit communication latency. This intelligent and constrained operator fusion mechanism, based on the aforementioned dynamic resource scheduling and pipeline construction, further optimizes data flow and improves the overall execution efficiency and throughput of edge-side multi-model collaborative inference.

[0111] The following example illustrates this. Assume that when building a system-level inference pipeline, there exists an upstream model node in the directed acyclic computation graph whose output operator is a ReLU activation function, followed by an input operator for a downstream model node, which is a Batch Normalization layer. The system first determines whether the output tensor of the ReLU operator (e.g., float32 type, dimension [1, 64, 56, 56]) matches the expected input tensor of the Batch Normalization operator (e.g., float32 type, dimension [1, 64, 56, 56]) in terms of data precision, data dimension, and computation type. If these attributes are highly matched, and the matching degree exceeds a preset fusion threshold (e.g., 90%), the system will fuse the ReLU operator and the Batch Normalization operator into a composite operator named "Fused_ReLU_BN". The scheduler then assigns this "Fused_ReLU_BN" composite operator to a specific heterogeneous computing unit (e.g., an NPU) for sequential execution. In this case, the ReLU calculation results can be directly passed to Batch Normalization for processing within the NPU, without having to write intermediate data to main memory or transfer it between the NPU and other computing units.

[0112] Through the above technical solution, this application can intelligently identify and fuse adjacent and compatible operators in a directed acyclic computation graph, and ensure that the fused composite operator is executed continuously within a single computational unit. This significantly reduces the number of times intermediate data is moved between memory and different heterogeneous computational units, thereby effectively reducing memory bandwidth usage and data communication latency between heterogeneous computational units. In complex edge-side multi-model collaborative inference scenarios, this optimization is of great significance for improving overall inference efficiency, reducing system power consumption, and enhancing real-time response capabilities.

[0113] Furthermore, this embodiment also maps the operators that undertake computational tasks in each AI model to the heterogeneous computing units corresponding to the global resource pool based on resource scheduling strategies, model dependencies, model nodes in the directed acyclic computation graph, and the fused operator types. It then uses pipelined parallel technology to execute the system-level inference task pipeline, enabling different AI models at different stages in the directed acyclic computation graph to process different batches or different parts of data simultaneously, thereby improving overall hardware utilization and system throughput.

[0114] Specifically, the operators responsible for computational tasks in each AI model are mapped to heterogeneous computing units corresponding to a global resource pool. This aims to precisely allocate the scheduled and fused computational tasks to the most suitable heterogeneous computing units for execution. This is not merely simple resource allocation, but a matching process based on the characteristics of the operators (such as computation type and data precision requirements) and the capabilities of the heterogeneous computing units (such as floating-point arithmetic, integer arithmetic, and memory bandwidth). For example, mapping can be achieved through a pre-established operator-hardware performance database, which records the execution efficiency, power consumption, and other metrics of different types of operators on different heterogeneous computing units (such as CPUs, GPUs, and NPUs). During runtime, the scheduler queries this database to select the computing unit with the best performance or lowest power consumption for each operator. Alternatively, heuristic algorithms or reinforcement learning-based methods can be used for dynamic mapping. The system continuously learns and adjusts the operator-to-heterogeneous computing unit mapping strategy during operation to adapt to constantly changing hardware states and model loads, thereby achieving better resource utilization and performance.

[0115] Meanwhile, employing pipelined parallelism to execute system-level inference task pipelines is a parallel computing paradigm that improves system throughput and hardware utilization by decomposing a large computational task into multiple stages and allowing these stages to process different data simultaneously on different processing units. Here, it is applied to the entire system-level inference task pipeline, meaning that different AI models or different stages of the same AI model can be executed concurrently on different heterogeneous computing units. For example, the entire directed acyclic computation graph can be divided into several logical stages, each containing one or more model nodes or fused operators. Once the first batch of data has been processed by the first stage, it immediately enters the second stage, while the first stage begins processing the second batch of data, and so on, forming parallel processing of the data stream. Alternatively, a pipeline manager can be implemented in the scheduler, responsible for maintaining the input / output buffers of each stage and coordinating the execution of operators on different heterogeneous computing units. When an operator completes its computation and writes the result to the output buffer, the pipeline manager notifies the operator of the next stage to begin execution, thus achieving seamless data flow between different computing units. This approach enables different AI models at different stages in a directed acyclic computation graph to process different batches or portions of data simultaneously, significantly improving overall processing efficiency. For example, for a continuously input data stream (such as video frames), each frame or several frames can be treated as a batch. When the first frame of data is processed in model A, the second frame can be processed in model B, or when the first frame is processed in the first layer of model A, the second layer can process the data from the previous frame. For a single large AI model, its internal computation graph can be divided into multiple subgraphs or layers, and these subgraphs or layers can be considered as different stages of a pipeline. For example, the first half of the model can be processed on the NPU, and the second half on the GPU. When the NPU has finished processing the first half of the first batch of data, the GPU can begin processing the second half of the first batch of data, while the NPU begins processing the first half of the second batch of data.

[0116] This application's solution further improves system performance by introducing a refined operator mapping and pipelined parallel execution mechanism, based on resource scheduling strategies, model dependencies, model nodes in the directed acyclic computation graph, and the fused operator types. The system first identifies the computational tasks (operators) to be executed in each AI model based on the previously determined resource scheduling strategy, combined with the dependencies between models and the structure of the directed acyclic computation graph. These operators may have been optimized through operator fusion technology to form composite operators. Subsequently, the system precisely maps each operator to the most suitable heterogeneous computing unit for execution based on the specific type of these operators, their computational requirements, and the real-time available resources and performance characteristics of each heterogeneous computing unit (such as CPU, GPU, NPU) in the global resource pool. This mapping considers the matching degree between the operator and the hardware, aiming to maximize the execution efficiency of a single operator. Furthermore, to fully utilize the parallel processing capabilities of multiple heterogeneous computing units on the edge device, the system employs pipelined parallel technology to execute the entire system-level inference task pipeline. This means that the entire inference process is broken down into multiple consecutive stages, each of which can consist of one or more AI models or their internal operators. Once the first batch of data has been processed through the first stage of the pipeline, it is immediately passed to the second stage for processing, while the first stage can begin processing the second batch of data. This data-stream parallel processing approach allows different AI models at different stages in the directed acyclic computation graph to process different batches or parts of data simultaneously. For example, while one AI model is processing the current frame in a video stream, another AI model can simultaneously process the recognition result of the previous frame, or different layers of the same AI model can process different parts of the same frame in parallel on different computational units. Through this fine-grained operator mapping and pipelined parallel execution, the system effectively avoids idle waiting times for computational units, significantly improving the overall utilization of heterogeneous computational units on the edge device, thereby greatly increasing the system throughput of multi-model collaborative inference without increasing additional hardware costs. This collaborative approach organically combines resource scheduling, model dependency handling, operator fusion, fine-grained hardware mapping, and pipelined parallel execution to form an efficient and adaptive edge-side multi-model inference framework. It effectively solves the problems of insufficient hardware resource utilization and low inference efficiency in complex multi-model scenarios.

[0117] The following is a concrete example to illustrate this. Assume two AI models are deployed on the edge device: one for object detection (Model A) and the other for object tracking (Model B), and the input of Model B depends on the output of Model A. When building the system-level inference task pipeline, Model A is first executed before Model B based on the model dependencies. During the operator mapping stage, computationally intensive convolution operators in Model A can be mapped to the NPU for execution, while some logical judgments or data post-processing operators in Model B can be mapped to the CPU for execution. When the execution pipeline is parallelized, when the first frame of the video stream enters Model A's NPU for object detection, the system does not wait for Model A to completely process the first frame. Once Model A's NPU completes the detection of the first frame and outputs the result, this result is immediately passed to Model B's CPU for object tracking. Simultaneously, Model A's NPU begins processing the second frame of the video stream. In this way, Model A and Model B execute concurrently, allowing the NPU and CPU to work simultaneously. Furthermore, if some layers within Model A can be parallelized—for example, the feature extraction layer executes on the NPU while the subsequent classification layer executes on the GPU—then after the NPU finishes processing the feature extraction of the first frame, the GPU can immediately begin processing the classification of the first frame, while the NPU begins processing the feature extraction of the second frame. In this way, different AI models or different parts of the same AI model can process different batches of data simultaneously, thereby achieving full utilization of hardware resources and a significant improvement in system throughput.

[0118] By employing the aforementioned technical solution, operators undertaking computational tasks in each AI model are precisely mapped to heterogeneous computing units corresponding to the global resource pool. Pipeline parallelism is then used to execute the system-level inference task pipeline, enabling different AI models at different stages in the directed acyclic computation graph to simultaneously process different batches or portions of data. This effectively solves the problem of low utilization of heterogeneous computing units and limited inference throughput that may still exist in edge-side multi-model collaborative inference, even with resource scheduling and operator fusion. Through refined operator-hardware matching and parallel data flow processing, the system can maximize the potential of heterogeneous computing units on edge devices, significantly improve overall hardware utilization, and substantially increase the system throughput of multi-model collaborative inference, thereby achieving more efficient and real-time AI inference services with limited edge resources.

[0119] In other embodiments, this application proposes an edge-side multi-model collaborative inference method. This method can dynamically collect hardware resource status data of edge devices, obtain static metadata and dynamic inference performance data of AI models, and dynamically allocate computing resources to AI models based on this data. It determines inference priorities and execution parallel strategies to generate resource scheduling strategies, and finally constructs and executes a system-level inference task pipeline based on the resource scheduling strategy and model dependencies to complete collaborative inference. However, in practical applications, although the system can perform automated scheduling and optimization, its complex internal decision-making processes and real-time operating status are often opaque to external operators. This lack of visibility and intervention capabilities may make it difficult for operators to monitor system performance in real time, promptly identify potential problems, or make fine-tuning adjustments according to specific scenario requirements, thereby affecting the overall efficiency and reliability of the system.

[0120] In this regard, this application further proposes that the method also includes providing a visual monitoring interface for real-time display of hardware resource status data, dynamic inference performance data of each AI model, inference priority, resource scheduling strategy, and execution status of the system-level inference task pipeline; and receiving instructions for external manual intervention through the visual monitoring interface, wherein the manual intervention includes: adjusting the inference priority of the AI ​​model, triggering or terminating operator fusion, and reallocating computing resources to the AI ​​model.

[0121] The visual monitoring interface is a graphical user interface designed to present complex system data in an intuitive and easy-to-understand way. Its function is to provide operators with a unified window to gain real-time insight into the operational status of the multi-model inference system. This interface can be implemented as a web-based dashboard accessible through a browser, as a dedicated application running on edge devices, or as a remote client application. It can utilize various visualization technologies, such as charts, graphs, dashboards, and status indicators, to display data. Real-time display ensures that the presented information reflects the current state of the system with minimal delay, supporting timely decision-making and problem detection. This can be achieved through continuous polling of data from system sensors and software modules, event-driven update mechanisms, or push notifications from the monitoring backend. The data refresh rate can be configured according to system criticality and resource availability. Hardware resource status data refers to metrics collected from heterogeneous computing units (such as CPUs, GPUs, and NPUs), including computing power utilization, memory usage, and power consumption, used to display the load and health status of the underlying hardware. Dynamic inference performance data for AI models includes actual latency, throughput, and resource utilization on heterogeneous computing units during the inference process, reflecting the real-time performance of each AI model. Inference priority is a parameter assigned to each AI model, indicating its relative importance or urgency in the scheduling mechanism; higher-priority models typically receive priority in resource allocation and execution order. The resource scheduling strategy is a comprehensive plan for allocating computing resources to all AI models, determining inference priorities, and defining parallel execution strategies. The execution status of the system-level inference pipeline provides information on the current execution stage of the entire inference pipeline, including which models are running, which are waiting, and the overall progress. Receiving instructions for external human intervention provides a mechanism allowing human operators to override or adjust automated system decisions based on their expertise or specific contextual needs. This can be achieved through input fields, buttons, sliders, or dropdown menus in a visual monitoring interface, which can trigger specific API calls or system commands. Adjusting the inference priority of AI models allows operators to manually increase or decrease the importance of certain models, for example, in response to critical events or changing application requirements. This can be achieved through numeric input fields, sliders, or selections of predefined priority levels associated with each model in the interface. Triggering or terminating operator fusion allows operators to manually control the optimization process of compatible operator combinations. This can be useful for debugging, performance tuning, or resolving compatibility issues. This can be achieved through toggles, checkboxes, or dedicated buttons in the interface, which send commands to the pipeline build modules. Reallocating computational resources to AI models allows operators to manually allocate or reallocate specific heterogeneous computing units or a certain amount of resources to AI models, thus overriding automatic dynamic allocation.This is crucial for troubleshooting or ensuring that critical models have dedicated resources. This can be achieved through drag-and-drop functionality in the interface, a menu for selecting available compute units, or an input field for the percentage of resources.

[0122] This application's solution significantly enhances the controllability and transparency of the edge-side multi-model collaborative inference system by introducing a visual monitoring interface and a manual intervention mechanism. The visual monitoring interface, as the core hub, aggregates and presents all key operational data in real time, including hardware resource status data, dynamic inference performance data of AI models, inference priorities, resource scheduling strategies, and the execution status of the system-level inference task pipeline. This comprehensive real-time visibility effectively solves the opacity problem in the automated scheduling process, enabling operators to clearly understand the system's internal operating logic and performance. Simultaneously, by providing manual intervention operations such as adjusting the inference priority of AI models, triggering or terminating operator fusion, and reallocating computing resources to AI models, this solution empowers operators to directly influence automated decisions. This means that when faced with unexpected situations, specific business needs, or high-level strategies that automated algorithms cannot fully capture, human operators can intervene promptly to make fine-tuning adjustments, thereby correcting potential suboptimal decisions and ensuring the system always operates in its optimal state. This combination of automated optimization and manual control enables the system to not only operate autonomously and efficiently, but also to intervene when necessary through human wisdom and experience, thereby significantly improving the system's robustness, adaptability, and reliability. It is especially suitable for edge-side application scenarios with extremely high requirements for real-time performance, accuracy, and stability.

[0123] The following is a concrete example. Assume an edge device deployed in an intelligent transportation system, running multiple AI models, such as those for real-time traffic flow analysis, anomaly detection, and traffic signal optimization. This device contains heterogeneous computing units such as CPUs and NPUs. During system operation, a visual monitoring interface displays the real-time computing power utilization, memory usage, and overall power consumption of each NPU core. It also shows the real-time inference latency, throughput, and resource consumption of the traffic flow analysis and anomaly detection models on the NPUs. The interface clearly indicates the inference priority of each model (e.g., the anomaly detection model has higher priority than the traffic flow analysis model) and the current resource scheduling strategy used by the system (e.g., the anomaly detection model exclusively uses NPU core 1, while the traffic flow analysis and traffic signal optimization models share NPU core 2). Furthermore, the execution status of the system-level inference task pipeline is displayed in the form of a flowchart, indicating the data flow and the current processing stage of each model. When operators at the traffic management center observe through a visual monitoring interface that a sudden surge in traffic volume during a specific period causes a significant increase in inference latency for the traffic flow analysis model, but the system's automatic scheduling fails to promptly upgrade its priority, they can manually adjust the model's inference priority from "medium" to "high" using interactive controls on the interface. This ensures the model receives more computing resources, reducing latency and enabling timely responses to traffic changes. Alternatively, if an operator detects a performance degradation trend in a particular NPU core, they can manually migrate an AI model running on that core to a healthy NPU core to avoid performance bottlenecks. Furthermore, during system debugging or the initial deployment of a new model, operators can manually trigger or terminate operator fusion between specific models to observe its impact on performance, allowing for more refined performance tuning.

[0124] Through the above technical solutions, operators can gain a real-time and intuitive understanding of the operational status and performance indicators of the edge-side multi-model collaborative inference system, enabling them to promptly identify and resolve potential problems. Simultaneously, the introduction of a manual intervention mechanism allows operators to flexibly adjust the model's inference priority, operator fusion strategy, and computing resource allocation according to actual needs or unforeseen circumstances. This effectively compensates for the limitations of purely automated scheduling in handling complex and ever-changing scenarios, significantly improving the system's controllability, adaptability, and reliability.

[0125] For example, to help understand the technical concept and principles of this embodiment, please refer to Figure 4 , Figure 4 This is a schematic diagram of the system architecture of the edge-side multi-model collaborative inference method according to an embodiment of this application, wherein: 1. Computing power monitoring module The computing power monitoring module requires three data acquisition phases to ensure the completeness of performance monitoring. The first phase is responsible for hardware resource acquisition: First, a dynamic acquisition cycle of 10ms-100ms is set, and the acquisition frequency is dynamically adjusted according to the device load to reduce the overhead of the monitoring module itself. Key indicators such as CPU / GPU / NPU utilization, memory usage, power consumption, temperature status, and bandwidth utilization are collected in real time. The methods for collecting these indicators are as follows: Real-time CPU / GPU / NPU load data is obtained through hardware driver interfaces such as Intel PowerGadget, NVIDIA SMI, and NPU vendor SDKs. Simultaneously, the system memory and video memory usage are monitored to avoid memory overflow or resource contention. Power consumption data is obtained through power management interfaces such as ACPI, PMU, or dedicated sensors to ensure the device operates within its thermal design power range. Core temperature is monitored to prevent performance throttling or hardware damage caused by overheating. Bandwidth utilization of data transmission channels such as PCIe and memory bus is monitored to ensure smooth data flow. The second phase is responsible for model registration and metadata management: When the system loads a model, metadata such as the model's computation graph structure, operator type, input / output tensor size, and precision requirements (FP32 / FP16 / INT8) are registered. The model is analyzed using frameworks such as ONNX / TensorRT / TVM to extract static computing power requirements (FLOPs or MACs). The third stage is responsible for dynamic computing power assessment: real-time tracking of inference latency, throughput, and resource consumption of registered models, and dynamic adjustment of model priority and resource allocation weights based on hardware monitoring data. In real-time system scenarios, if a model's latency exceeds the limit and the total GPU utilization has not reached the upper limit, a signal for resource reallocation is triggered and sent to the subsequent dynamic scheduling module, allowing the model to obtain more computing resources and reduce latency.

[0126] 2. Dynamic scheduling module The dynamic scheduling module intelligently adjusts the inference priority, parallelism, and resource allocation strategies of each model based on the real-time hardware resource status and model computing power requirements collected by the computing power monitoring module. This module employs a multi-layered scheduling architecture. First, it manages heterogeneous computing resources such as CPUs, GPUs, and NPUs through a global resource pool, and then dynamically allocates computing resources according to the model's real-time load and dependencies. For example, for models with high computing power requirements, such as text generation, the system prioritizes allocating more GPU cores or NPU acceleration units and ensures low-latency execution of critical tasks through priority queues. Conversely, for models with lower computing power requirements, such as object detection, resource consumption is appropriately reduced to avoid resource waste. To achieve efficient dynamic scheduling, the module incorporates an adaptive load balancing algorithm that combines historical inference data and real-time monitoring feedback to predict the changing trends of model computing power requirements and adjust resource allocation in advance. For example, when a sudden increase in input data traffic for a model is detected, the system automatically triggers a resource pre-allocation mechanism to avoid inference latency caused by resource contention.

[0127] 3. Collaborative Reasoning Module The collaborative reasoning module is configured to build and execute a system-level reasoning task pipeline based on resource scheduling strategies and model dependencies.

[0128] The collaborative inference module is responsible for transforming the results of dynamic scheduling into an efficient inference execution flow. Through model dependency analysis and data flow optimization, it achieves end-to-end collaborative inference. The entire module is based on the Foster design methodology, automating and integrating the original four-step parallel algorithm design—partitioning, communication, aggregation, and mapping—into the module. First, it combines the computational graph structure and operator types of each model obtained from the computational power monitoring module to break down all models in the system. Based on the dependencies between models, it constructs a new computational graph representing the execution order of inference tasks, thus determining how each unit communicates. Then, a model fusion unit within the module detects the operator matching degree of adjacent models in the computational graph and performs automated operator fusion, reducing the number of times data is moved between different hardware. Finally, it combines the resources allocated by the dynamic scheduling module to perform resource mapping in the fusion module, automatically generating the optimal inference pipeline. For example, for a multi-stage model that includes image preprocessing, object detection, feature extraction, and semantic retrieval, the system allocates preprocessing tasks to the CPU, object detection and feature extraction tasks to the GPU, and semantic retrieval tasks are dynamically selected to be executed by the NPU or CPU based on the real-time load to maximize resource utilization. To further improve inference efficiency, the collaborative inference module adopts pipelined parallelism, which allows different stages of multiple models to be executed simultaneously. For example, while one batch of image data is entering the feature extraction stage, the next batch of data has already begun preprocessing, thereby reducing resource idle waiting time.

[0129] Finally, the collaborative inference module provides a visual monitoring interface that displays the execution status, resource consumption, and latency metrics of the inference flow in real time, and supports manual intervention, such as forcibly terminating low-priority tasks or reallocating resources. Through closed-loop feedback with the dynamic scheduling module, the system can continuously optimize the inference process, adapt to changes in computing power requirements under different scenarios, and achieve efficient and stable operation of the edge intelligent system.

[0130] in, Figure 4 The invention clearly demonstrates the "perception-scheduling-inference-feedback" closed-loop optimization mechanism formed by its method. Performance metrics generated from each round of inference execution are fed back to update the model's cognition and optimization algorithms (such as adjusting the internal weight parameters of the adaptive load balancing algorithm), enabling the system to continuously adapt to load changes.

[0131] Taking a multi-model application that includes image preprocessing, object detection, and feature extraction as an example: the system may divide the tasks of preprocessing (CPU), object detection (GPU), and feature extraction (NPU) and construct a directed acyclic computation graph based on dependencies; the dynamic scheduling module dynamically allocates computing resources for each stage and determines priorities based on real-time load; the collaborative inference module may fuse compatible operators in detection and extraction and organize them into a pipeline for parallel execution. The entire process can be observed through a visual monitoring interface, and subsequent scheduling can be automatically optimized based on feedback data.

[0132] Compared with the prior art, this embodiment has the following three beneficial effects: First, it has a wide range of applications. This embodiment is not targeted at any specific platform or hardware, but is adapted to any intelligent system on the edge, providing a unified and complete universal solution. Through the computing power monitoring module, it simultaneously collects hardware resource and model resource information, thereby achieving true algorithm-hardware co-design.

[0133] Second, the optimization scale is flexible. This embodiment integrates multiple system design concepts such as load balancing, task scheduling, and pipeline design, and introduces the Foster design method, thus combining the concurrency concept of the software layer and the parallel concept of the hardware layer. It opens up the optimization scale and granularity of different levels from top to bottom, achieving the ultimate optimization of the entire system, and is perfectly applicable to resource-constrained and power-sensitive edge device scenarios.

[0134] Third, adaptive adjustment and improvement. Currently, most inference optimization is completed offline and then redeployed. This inevitably leads to the loss of some boundary conditions of the online scenario during inference optimization. Although iterative optimization can be performed manually, it obviously increases labor costs. This embodiment incorporates the backpropagation method from model training into online inference using the concept of dynamic computing power awareness. For each round of the inference pipeline, performance results are fed back to the system front end, thereby dynamically adjusting the system's resource allocation and task scheduling.

[0135] Fourth, in addition to the adaptive function brought by computing power awareness, alternative manual intervention schemes have also been introduced in the collaborative reasoning module. Through the visual monitoring interface, staff can observe the reasoning situation of the entire system in real time and make selective adjustments based on monitoring indicators and computing power intensity heatmaps.

[0136] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multi-model collaborative reasoning method on the end side of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0137] This application provides an edge device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to execute the edge multi-model collaborative reasoning method in the first embodiment described above.

[0138] The following is for reference. Figure 5 It shows a schematic diagram of the structure suitable for implementing the end-side device of the embodiments of this application. Figure 5 The end-side device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0139] like Figure 5As shown, the edge device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the edge device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the end-side device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show end-side devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0140] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0141] The edge device provided in this application adopts the edge-side multi-model collaborative inference method in the above embodiments, which can solve the technical problems of low resource utilization and insufficient inference efficiency in edge-side multi-model inference schemes. Compared with the prior art, the beneficial effects of the edge device provided in this application are the same as the beneficial effects of the edge-side multi-model collaborative inference method provided in the above embodiments, and other technical features in this edge device are the same as the features disclosed in the edge-side multi-model collaborative inference method of the previous embodiment, and will not be repeated here.

[0142] It should be understood that the various parts disclosed in the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0143] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0144] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the edge-side multi-model collaborative reasoning method in the above embodiments.

[0145] The storage medium provided in this application embodiment is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described edge-side multi-model collaborative inference method. This solves the technical problems of low resource utilization and insufficient inference efficiency in edge-side multi-model inference schemes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the edge-side multi-model collaborative inference method provided in the above embodiments, and will not be repeated here.

[0146] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An edge-side multi-model collaborative inference method, applied to an edge device containing multiple heterogeneous computing units, wherein multiple AI models are deployed on the edge device, characterized in that, include: The hardware resource status data of the end-side device is dynamically collected. The hardware resource status data includes at least the computing power utilization rate, memory occupancy rate and operating power consumption of each heterogeneous computing unit. Obtain static metadata and dynamic inference performance data for each AI model. The static metadata includes the computation graph structure, operator type, and input / output tensor dimensions parsed from the AI ​​model. The dynamic inference performance data includes the actual latency, throughput, and resource utilization rate on each heterogeneous computing unit during the model inference process of the AI ​​model. Based on the hardware resource status data, the static metadata, and the dynamic inference performance data, computing resources of the heterogeneous computing units are dynamically allocated to each AI model, and the inference priority and execution parallel strategy of each AI model are determined to generate a resource scheduling strategy. The execution parallel strategy is used to indicate whether models without dependencies are executed in parallel and whether a single model is computed in parallel on multiple computing units. The inference priority and execution parallel strategy need to be decided in combination with the model dependencies between each AI model. Based on the resource scheduling strategy and the model dependencies, a system-level inference task pipeline is constructed and executed to complete collaborative inference. The construction of the system-level inference task pipeline includes: determining the execution order based on the model dependencies, constructing a directed acyclic computation graph representing the data flow between models, and performing operator fusion on adjacent and operator-compatible model nodes in the directed acyclic computation graph.

2. The end-side multi-model collaborative inference method as described in claim 1, characterized in that, After the step of constructing and executing a system-level inference task pipeline to complete collaborative inference based on the resource scheduling strategy and the model dependencies, the method further includes: The system collects performance metrics during the execution of the system-level inference task pipeline and feeds these performance metrics back to update the dynamic inference performance data in order to optimize the resource scheduling strategy and inference task pipeline for the next round.

3. The end-side multi-model collaborative inference method as described in claim 1 or 2, characterized in that, The step of dynamically collecting hardware resource status data of the terminal device includes: The hardware resource status data is collected at a dynamically adjustable collection period; The collection cycle is adjusted based on the current system load or the urgency of the AI ​​model's inference task. When the system load is higher than a preset threshold or there are tasks with an inference task urgency greater than a preset value, the collection cycle is shortened to increase the monitoring frequency. When the system load is lower than a preset threshold and there are no tasks with an inference task urgency greater than a preset value, the collection cycle is extended to reduce system overhead.

4. The end-side multi-model collaborative inference method as described in claim 2, characterized in that, The plurality of heterogeneous computing units include at least two of CPU, GPU, and NPU. The step of dynamically allocating computing resources to each of the AI ​​models based on the hardware resource status data, the static metadata, and the dynamic inference performance data includes: Establish and maintain a global resource pool to perform unified abstraction, quantification, and state management of at least two types of computing resources, namely CPU, GPU, and NPU. Based on the hardware resource status data, the static metadata, and the dynamic inference performance data, and combined with the real-time available resources of the global resource pool, computing resources are dynamically allocated to each of the AI ​​models.

5. The end-side multi-model collaborative inference method as described in claim 4, characterized in that, The steps for determining the inference priority of each model and the execution parallel strategy include: Based on the historical sequence of the dynamic inference performance data, an adaptive load balancing algorithm is used to predict the changes in computing power demand of each AI model in the next time window. By combining the predicted changes in computing power demand, the model dependencies, and the preset task urgency indicators, the inference priority of each model is calculated and dynamically adjusted.

6. The end-side multi-model collaborative inference method as described in claim 5, characterized in that, The step of feeding back the performance metrics to update the dynamic inference performance data in order to optimize the resource scheduling strategy for the next round includes: The actual latency and throughput collected are compared with the expected performance under the current resource scheduling strategy to generate prediction error; Based on the prediction error, an online learning mechanism is used to adjust the internal weight parameters of the adaptive load balancing algorithm used to predict changes in computing power demand.

7. The end-side multi-model collaborative inference method as described in claim 5, characterized in that, The method also includes a fault recovery mechanism: Real-time monitoring of the health status of each heterogeneous computing unit; When a fault or severe performance degradation is detected in the heterogeneous computing unit where the computing resources allocated to execute any AI model are located, the affected model inference task is migrated to other available heterogeneous computing units for continued execution based on the real-time available resources of the global resource pool, or the affected model inference task reduces its inference accuracy to reduce computing power requirements.

8. The end-side multi-model collaborative inference method as described in claim 7, characterized in that, The step of operator fusion for adjacent and operator-compatible model nodes in the directed acyclic computation graph includes: Determine whether the output operator of the upstream model node in the directed acyclic computation graph matches the input operator of the downstream model node in the directed acyclic computation graph in terms of data precision, data dimension, and computation type. If the matching degree exceeds the preset fusion threshold, the output operator and the input operator are fused into a composite operator. This composite operator is executed continuously within a computing unit to reduce the number of times intermediate data is moved between memory and different heterogeneous computing units.

9. The end-side multi-model collaborative reasoning method as described in claim 8, characterized in that, The steps of constructing and executing the system-level inference task pipeline also include: Based on the resource scheduling strategy, the model dependencies, the model nodes in the directed acyclic computation graph, and the fused operator types, the operators undertaking computational tasks in each AI model are mapped to the heterogeneous computing units corresponding to the global resource pool. Pipeline parallel technology is then used to execute the system-level inference task pipeline, enabling different AI models at different stages in the directed acyclic computation graph to process different batches or different parts of data simultaneously, thereby improving overall hardware utilization and system throughput.

10. The end-side multi-model collaborative inference method as described in claim 1 or 2, characterized in that, The method further includes: A visual monitoring interface is provided to display in real time the hardware resource status data, the dynamic inference performance data of each AI model, the inference priority, the resource scheduling strategy, and the execution status of the system-level inference task pipeline; The system receives instructions for external manual intervention through the visual monitoring interface. These manual interventions include: adjusting the inference priority of the AI ​​model, triggering or terminating operator fusion, and reallocating computing resources to the AI ​​model.

11. An end-side device, characterized in that, The edge device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the edge multi-model collaborative reasoning method as described in any one of claims 1 to 10.

12. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the edge-side multi-model collaborative reasoning method as described in any one of claims 1 to 10.

Images