A router gateway camera cooperates with AI inference acceleration optimization method

Abstract

The application provides a router gateway camera collaborative AI inference acceleration optimization method in the technical field of edge computing and artificial intelligence acceleration, and comprises the following steps: after the edge device is started, the local AI acceleration hardware is detected and reported to the router, and the router aggregates and constructs a heterogeneous computing resource pool view; after receiving an AI model, the AI model is analyzed and decomposed into subgraphs, a cost model is established, an optimal mapping scheme is searched in combination with the resource view, and a cross-device collaborative execution plan is generated; when responding to an inference request, the task is split into multiple stages according to the plan, and is executed in parallel in a pipeline; during execution, tensor data is directly transmitted between different devices through a zero-copy data channel; meanwhile, the hardware state is monitored, dynamic voltage frequency adjustment and task migration are performed, and abnormal migration is performed to standby hardware or CPU. The application has the advantages that the global acceleration ratio and resource utilization of AI inference tasks between heterogeneous edge devices are greatly improved, and system energy efficiency, real-time performance and scalability are considered.

Description

Technical Field

[0001] This invention relates to the field of edge computing and artificial intelligence acceleration technology, and in particular to a method for accelerating and optimizing AI inference through router gateway camera collaboration. Background Technology

[0002] With the widespread application of deep learning models in edge computing scenarios, accelerating artificial intelligence (AI) inference has become a core technological requirement for edge intelligent systems. In typical application scenarios such as smart homes, security monitoring, and industrial vision, distributed edge computing systems composed of cameras, gateways, and routers need to process multiple high-definition video streams in real time and perform computationally intensive AI inference tasks such as object detection, facial recognition, and behavior analysis. To address these needs, current mainstream edge devices generally integrate dedicated AI acceleration hardware, including neural network processing units (NPUs), graphics processing units (GPUs), digital signal processors (DSPs), and field-programmable gate arrays (FPGAs), aiming to improve local AI inference performance and reduce latency.

[0003] However, existing technologies have significant shortcomings in utilizing the aforementioned heterogeneous acceleration hardware. Currently, the mainstream solutions on the market mainly include the following: (i) Single-device independent acceleration mode. In this mode, each edge device only uses its own integrated AI acceleration hardware to complete inference tasks, and there is no computing power sharing or collaboration mechanism between cameras, gateways, and routers. The limitations of this mode are: it is limited by the computing power ceiling of a single device and cannot handle complex AI models (such as large-scale models like YOLOv8-X) that exceed local computing capabilities; at the same time, the problem of uneven load between devices is prominent, with some devices having overloaded computing resources while others are idle, resulting in low overall system resource utilization.

[0004] (ii) Cloud offloading acceleration mode. When local computing power is insufficient, AI inference tasks are offloaded to the cloud GPU cluster for execution. This solution relies on high bandwidth and low latency network conditions. However, the transmission overhead from the edge to the cloud is usually 50ms to 200ms, which is difficult to meet the application scenarios with high real-time requirements, and the service availability cannot be guaranteed when the network is unstable.

[0005] (iii) Static Hardware Binding Mode. This mode maps the model to specific hardware during the model deployment phase (e.g., fixing the NPU of a neural network model to a specific device), without making dynamic adjustments during runtime. The drawback of this approach is that it cannot adapt to dynamic fluctuations in hardware load (e.g., routers may be preempted due to a sudden increase in network traffic), nor can it utilize the idle computing power of other devices for task acceleration, resulting in unstable system performance when the load changes.

[0006] (iv) Homogeneous Cluster Acceleration Mode. This mode uses homogeneous hardware (such as NPUs of the same model in multiple cameras) to process inference tasks in parallel. This mode requires a highly consistent hardware architecture and cannot adapt to the inherent heterogeneity of edge devices. Cameras are usually equipped with lightweight NPUs, gateways are configured with mid-range GPUs, and routers integrate high-performance NPUs. The architectures of different devices are very different, the software stacks are incompatible, and there is a lack of cross-architecture computing power abstraction and unified scheduling capabilities.

[0007] In summary, the shortcomings of the existing technologies can be categorized into the following dimensions: First, heterogeneous computing power becomes isolated. AI acceleration hardware from different manufacturers differs fundamentally in instruction set architecture, memory model, supported operator types, and precision formats. Furthermore, their software stacks are incompatible, making it impossible to share and aggregate computing resources across devices. As a result, the overall system acceleration efficiency is typically less than 40%.

[0008] Second, the model and hardware matching is rigid. The static binding strategy adopted during the model deployment phase cannot be dynamically adjusted according to the real-time status of the hardware (including utilization, temperature, and power consumption) and the quality of service (QoS) requirements of the task. This results in drastic fluctuations in hardware utilization and inference latency jitter exceeding 30%, making it difficult to meet deterministic latency requirements.

[0009] Third, the task scheduling granularity is coarse. Existing solutions use complete models or complete video frames as scheduling units, which cannot perform model layering and pipeline parallelism based on network bandwidth conditions and the computing characteristics of each hardware. A single inference task cannot take advantage of the aggregated computing power of multiple devices to gain acceleration benefits.

[0010] Fourth, energy efficiency optimization is lacking. Existing solutions do not fully consider the power consumption characteristics of AI acceleration hardware and the power supply constraints of devices (such as battery-powered camera devices). Under high load operation, the devices are prone to overheating, frequency reduction, or power failure, resulting in poor system reliability.

[0011] Fifth, poor scalability and compatibility. New hardware integration requires the redevelopment of driver adaptation layers and interface conversion logic. It cannot automatically identify the characteristics of newly added acceleration units and incorporate them into a unified scheduling system, resulting in high system upgrade and maintenance costs and long cycles.

[0012] Therefore, how to provide a method for accelerating AI inference through router, gateway, and camera collaboration, thereby improving the global speedup and resource utilization of AI inference tasks among heterogeneous edge devices, while also taking into account system energy efficiency, real-time performance, and scalability, has become an urgent technical problem to be solved. Summary of the Invention

[0013] The technical problem to be solved by the present invention is to provide a method for accelerating and optimizing AI inference in a router-gateway-camera collaboration, thereby improving the global acceleration ratio and resource utilization of AI inference tasks among heterogeneous edge devices, while taking into account system energy efficiency, real-time performance and scalability.

[0014] This invention provides a method for accelerating AI inference through router gateway-camera collaboration, comprising the following steps: Step S1: After the edge device starts up, it automatically detects the local AI acceleration hardware and collects hardware specification information through the unified driver interface, and reports it to the router, which serves as the global scheduling center. The edge device includes the router, which serves as the global scheduling center, at least one gateway, and at least one camera under each gateway. The router summarizes the hardware specification information reported by each device and constructs a heterogeneous computing power resource pool view containing the AI ​​acceleration hardware and its hardware specification information of each edge device. After the camera collects the video stream, it initiates an AI inference request. Step S2: In response to the router receiving the AI ​​model, the AI ​​model is decomposed into multiple independently executable subgraphs based on computation graph analysis technology, and a computation cost model for each subgraph is established; combined with the hardware specification information of each AI acceleration hardware in the heterogeneous computing power resource pool view, the optimal subgraph-hardware mapping scheme that satisfies the latency constraint is searched, and a cross-device collaborative execution plan is generated. The cross-device collaborative execution plan includes subgraph partitioning, device mapping, data dependency, and synchronization point information. Step S3: In response to the AI ​​inference request, the AI ​​inference task corresponding to the AI ​​inference request is divided into multiple stages according to the cross-device collaborative execution plan, and mapped to the AI ​​acceleration hardware of different edge devices for pipelined parallel execution; during the execution process, the task arrival rate is monitored and the batch size is dynamically adjusted to aggregate multiple requests to form the optimal batch. Step S4: During the parallel execution of the pipeline, tensor data is directly transferred between the AI ​​acceleration hardware of different edge devices through the zero-copy data channel established between each edge device, thereby bypassing the memory transfer of the central processing unit. Step S5: Monitor the operating status of each AI acceleration hardware in real time, and perform dynamic voltage and frequency adjustment and task migration coordination operation according to the operating status; establish the power consumption-performance model of the AI ​​acceleration hardware to predict the risk of overheating or power failure; when hardware abnormality or performance degradation is detected, migrate the currently running AI inference task to the backup hardware or switch to the central processing unit for execution; when the hardware failure is recovered, automatically perform load rebalancing operation to migrate the AI ​​inference task back to the original hardware to restore system performance.

[0015] Furthermore, in step S1, the unified driver interface shields the differences in instruction set architecture, memory model, operator type, and precision format between AI acceleration hardware from different manufacturers, and provides a standardized computing power service interface to the upper layer; the hardware specification information includes computing power, memory capacity, supported operator types, and precision formats.

[0016] Furthermore, in step S1, the construction of the heterogeneous computing power resource pool view supports the automatic discovery of newly connected devices: when a new edge device is connected to the system, its AI acceleration hardware is automatically identified, and the hardware specification information is automatically registered to the resource pool.

[0017] Furthermore, in step S2, the computational cost model is established based on the number of floating-point operations and memory access patterns of the subgraph; the search for the optimal subgraph-hardware mapping scheme is implemented using dynamic programming or reinforcement learning algorithms.

[0018] Furthermore, step S2 also includes: before generating the cross-device collaborative execution plan, performing operator fusion, precision conversion, and memory layout optimization operations for different AI acceleration hardware; wherein, the operator fusion includes merging convolutional layers, batch normalization layers, and activation function layers; and the precision conversion includes adaptive selection of FP32, FP16, and INT8 precision formats.

[0019] Furthermore, in step S3, the multiple stages of the AI ​​inference task include a preprocessing stage, a feature extraction stage, and a post-processing stage; the pipeline parallel execution includes: the preprocessing stage and / or feature extraction stage being executed by the neural network processing unit of the camera, the feature fusion operation in the feature extraction stage and / or the post-processing stage being executed by the neural network processing unit or graphics processing unit of the gateway, and the detection output operation in the post-processing stage being executed by the high-performance neural network processing unit of the router. The dynamic batch size adjustment includes: monitoring the request queue used to cache the AI ​​inference requests; increasing the batch size when the request queue accumulates to improve hardware utilization; and decreasing the batch size when the request queue is cleared to reduce latency. The dynamic batch processing supports cross-device collaboration, whereby the gateway aggregates feature data from multiple cameras under its jurisdiction to form a large batch and sends it to the router for processing.

[0020] Furthermore, in step S4, the zero-copy data channel is implemented based on shared memory, remote direct memory access, or a direct memory access engine, and is used to directly exchange tensor data between AI acceleration hardware on different edge devices.

[0021] Furthermore, in step S5, the operating state includes temperature, power consumption, and power supply status; the dynamic voltage and frequency adjustment and task migration coordinated operation includes: prioritizing the use of the AI ​​acceleration hardware with the highest energy efficiency ratio under high load, and switching to low power mode or migrating the AI ​​inference task to backup hardware or central processing unit for execution under low load. The hardware anomaly or performance degradation includes hardware overheating and frequency reduction, hardware failure, and switching from mains power failure to battery power; the method also includes: when the battery power of the battery-powered device is detected to be lower than the power threshold, dynamically adjusting the inference accuracy and inference frequency of the AI ​​inference task according to the remaining power.

[0022] Furthermore, the method also includes: Step S6: Periodically evaluate the actual execution efficiency and energy consumption ratio of each AI acceleration hardware, and adjust the subgraph partitioning boundary and device mapping strategy online; predict the future load based on the spatiotemporal distribution pattern of task arrival, and perform pre-migration operations for AI inference tasks based on the prediction results; The periodic evaluation period is 5 seconds; the online adjustment strategy for subgraph partitioning boundaries and device mapping includes: when it is detected that the actual execution efficiency of a certain operator on the current neural network processing unit is lower than the expected efficiency threshold, the subgraph corresponding to the operator is dynamically migrated to the graphics processing unit for execution.

[0023] Furthermore, the router serves as a global computing power scheduling center and model optimization center, connecting to at least one gateway to form a star topology; the gateway serves as a regional computing power aggregation point, connecting to at least one subordinate camera to perform regional task scheduling and data aggregation; the camera serves as a data acquisition and front-end inference node, used to perform video stream acquisition and basic AI inference tasks. The router and the gateway are connected via 10 Gigabit Ethernet or fiber optic cable; the gateway and the camera are connected via Gigabit Ethernet or Power over Ethernet; the zero-copy data channel is established between each edge device through a high-speed interconnect channel, which includes PCIe, Ethernet and Wi-Fi 6.

[0024] The advantages of this invention are: 1. First, a unified driver interface is used to shield the underlying differences between AI acceleration hardware from different manufacturers. Specifications such as computing power, memory, and operator support for each device are automatically collected, and the router aggregates this information to generate a unified view of the heterogeneous computing resource pool, transforming isolated computing resources into a pooled system. Based on this, the input AI model is broken down into multiple independently executable subgraphs, and a computational cost model is established. The optimal subgraph-hardware mapping scheme is dynamically searched using resource pool information to generate a cross-device collaborative execution plan. The inference task is then distributed in a pipelined parallel manner across the camera NPU, gateway GPU, and router's high-performance NPU, according to preprocessing, feature extraction, and post-processing stages. Simultaneously, the batch size is dynamically adjusted based on the request queue length, and the gateway supports aggregating feature data from multiple cameras to form large batches, thereby fully utilizing the idle computing power of each device and avoiding single-point bottlenecks. During execution, devices directly communicate via shared memory or zero-copy data channels such as RDMA. By transferring tensor data and bypassing CPU-memory intermediaries, the system significantly reduces data transfer latency and CPU load, ensuring real-time performance. Simultaneously, the system continuously monitors the temperature, power consumption, and power supply status of each hardware component, establishing a power-performance model to predict overheating or power failure risks. It also collaboratively performs dynamic voltage and frequency adjustments and task migration, automatically switching tasks to backup hardware or the CPU during load fluctuations or anomalies, and automatically rebalancing after fault recovery. Furthermore, for battery-powered devices, it dynamically adjusts inference accuracy and frequency based on remaining power, thereby optimizing energy efficiency and reliability while ensuring performance. In addition, new devices are automatically identified and registered in the resource pool upon integration. Combined with periodic online evaluations of the actual execution efficiency and energy consumption ratio of each hardware component, the system dynamically adjusts subgraph partitioning boundaries and mapping strategies, and performs task pre-migration based on load prediction, enabling continuous self-optimization. Ultimately, this improves global inference throughput and resource utilization efficiency while also considering deterministic latency, system energy efficiency, and scalability.

[0025] 2. By setting a unified driver interface, the differences in instruction set architecture, memory model, operator type, and precision format of AI acceleration hardware from different manufacturers are shielded at the driver level, providing a standardized computing power service interface to the upper layer. At the same time, the router, as a global scheduling center, can automatically detect newly connected AI acceleration hardware and automatically register its hardware specifications to the heterogeneous computing power resource pool. This mechanism realizes unified management and plug-and-play functionality of heterogeneous computing power resources, significantly reducing the adaptation cost during system expansion and greatly improving the maintainability and hardware compatibility of the system in long-term operation.

[0026] 3. By employing computation graph analysis technology, the AI ​​model is decomposed into multiple independently executable subgraphs, and a computational cost model including floating-point operation counts and memory access patterns is established. Based on this, combined with the real-time specifications of each hardware in the resource pool, dynamic programming or reinforcement learning algorithms are used to search for the optimal subgraph-hardware mapping scheme that satisfies latency constraints in the global space. This approach refines the scheduling granularity from the "model level" to the "operator level subgraph," enabling a single inference task to be decomposed and pipelined to various hardware on different devices for parallel execution. This effectively breaks through the single-device computing power ceiling and significantly improves the overall inference throughput of the system.

[0027] 4. Based on the cross-device collaborative execution plan, the AI ​​inference task is divided into multiple stages such as preprocessing, feature extraction, and postprocessing, and is differentiated and mapped to the camera NPU, gateway GPU / NPU, and router high-performance NPU to form a parallel physical pipeline across devices. At the same time, the batch size of the request queue is dynamically adjusted by monitoring the task arrival rate in real time—increasing the batch size when the queue is backlogged to improve hardware utilization, and decreasing the batch size when the queue is empty to reduce response latency. This dynamic batch processing also supports the gateway to aggregate feature data from multiple cameras under its jurisdiction to form a large batch and send it to the router in a unified manner. This mechanism enables the system to maintain high throughput while strictly controlling tail latency when processing multiple burst video streams, achieving an excellent balance between throughput and latency.

[0028] 5. By establishing zero-copy data channels between edge devices based on shared memory, remote direct memory access, or direct memory access engines, tensor data can be directly transferred between AI acceleration hardware on different devices, completely bypassing the memory transfer of the central processing unit. This solution reduces the transmission latency of intermediate feature data across devices to the microsecond level, fundamentally eliminating the additional overhead caused by traditional data copying. It provides key low-latency communication guarantees for the engineering implementation of fine-grained pipeline parallelism, ensuring the real-time performance of the entire collaborative inference system.

[0029] 6. A power consumption-performance model for AI acceleration hardware has been established to monitor temperature, power consumption, and power supply status in real time, and to perform coordinated operations of dynamic voltage and frequency adjustment and task migration. When abnormal situations such as hardware overheating and frequency reduction, hardware failure, or power outage and switching to battery power are detected, the currently running inference tasks are automatically migrated to backup hardware or switched to the central processing unit for execution. After the fault is recovered, the load rebalancing is automatically performed to migrate the tasks back to restore optimal performance. For battery-powered devices, the inference accuracy and frequency can also be dynamically adjusted according to the remaining power. This mechanism significantly improves the reliability and energy efficiency of the system under complex power supply and heat dissipation conditions, effectively prevents service interruptions caused by device overheating and power failure, and extends the continuous working time of battery-powered devices.

[0030] 7. An online closed-loop optimization mechanism is introduced to periodically (e.g., every 5 seconds) evaluate the actual execution efficiency and energy consumption ratio of each AI acceleration hardware, and dynamically adjust the subgraph partitioning boundary and device mapping strategy accordingly. For example, when the execution efficiency of a certain operator on the current NPU is detected to be lower than the expected threshold, its corresponding subgraph is automatically migrated to the GPU for execution. At the same time, the system predicts future load based on the spatiotemporal distribution pattern of task arrival and performs pre-migration operations for inference tasks in advance. This adaptive capability enables the system to continuously approach the globally optimal computing power allocation strategy as the business load and hardware status change dynamically, significantly improving the average speedup ratio and overall energy efficiency of the system during long-lifecycle operation. Attached Figure Description

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] Figure 1 This is a flowchart of a router gateway camera collaborative AI inference acceleration optimization method according to the present invention. Detailed Implementation

[0033] The overall approach of the technical solution in this application is as follows: Starting from the pain points of collaborative utilization of heterogeneous AI acceleration hardware (such as camera NPUs, gateway GPUs / NPUs, and router high-performance NPUs) in edge computing scenarios, it first achieves automatic discovery, abstraction, and pooling of heterogeneous computing power through a unified driver interface, breaking down computing power silos. Then, for the input AI model, it employs computation graph decomposition and cost modeling, combined with a global resource view to dynamically search for the optimal subgraph-hardware mapping scheme, generating a cross-device collaborative execution plan. When an inference request arrives, the task is divided into multiple stages (such as preprocessing, feature extraction, and post-processing) according to this plan, and executed in a cross-device pipeline parallel manner. Simultaneously, the batch size is dynamically adjusted based on the request queue length to balance throughput and latency. During execution, it utilizes shared memory or RDMA, etc. Zero-copy data channels enable direct transfer of tensor data between devices, bypassing CPU-memory relay and ensuring low-latency communication. In addition, the system monitors the temperature, power consumption, and power supply status of each hardware component in real time, predicts overheating or power failure risks through a power consumption-performance model, and coordinates dynamic voltage and frequency adjustment and task migration. In case of anomalies, it automatically switches to backup hardware or CPU and automatically rebalances after fault recovery. For battery-powered devices, it can also dynamically adjust inference accuracy and frequency based on remaining power. Finally, by periodically evaluating hardware execution efficiency and energy efficiency ratio, it adjusts subgraph partitioning and mapping strategies online and implements task pre-migration in conjunction with load prediction, forming a continuously self-optimizing closed-loop control. This significantly improves the global speedup and resource utilization of AI inference tasks among heterogeneous edge devices, while taking into account energy efficiency, real-time performance, and system scalability.

[0034] Please refer to Figure 1As shown, a preferred embodiment of the router gateway camera collaborative AI inference acceleration optimization method of the present invention includes the following steps: Step S1: After the edge device starts up, it automatically detects the local AI acceleration hardware and collects hardware specification information through the unified driver interface, and reports it to the router, which serves as the global scheduling center. The edge device includes the router, which serves as the global scheduling center, at least one gateway, and at least one camera under each gateway. The router summarizes the hardware specification information reported by each device and constructs a heterogeneous computing power resource pool view containing the AI ​​acceleration hardware and its hardware specification information of each edge device. After the camera collects the video stream, it initiates an AI inference request. In practice, the unified driver interface is implemented as a service process running in the kernel or user space of the operating system of each edge device. This interface encapsulates vendor-specific drivers and runtime libraries for different manufacturers' AI acceleration hardware (such as Huawei Ascend NPU, NVIDIA GPU, ARM Ethos NPU, etc.), exposing a set of standardized function call interfaces to the upper-layer AI inference framework, such as query_device_info(), allocate_tensor(), and execute_operator().

[0035] When an edge device powers on, the unified driver interface automatically enumerates the AI ​​acceleration devices mounted on the local hardware bus (such as PCIe, AMBA) and obtains hardware specification information by reading the device's read-only registers or calling the manufacturer's API. This hardware specification information includes at least: computing power (unit: TOPS or FLOPS), on-chip memory capacity (unit: MB or GB), a list of supported operators (such as convolution, pooling, fully connected, activation functions, etc.), and supported precision formats (such as FP32, FP16, BF16, INT8, INT4). After acquisition, each edge device encapsulates the above information into standardized messages and reports them to the router's resource management module via a secure transmission protocol (such as TLS).

[0036] After receiving the hardware specification information reported by all edge devices, the router maintains a global resource table in memory. This table, indexed by device ID, records each device's device type (router / gateway / camera), hardware model, specifications, current available computing power, and network address. At this point, the heterogeneous computing power resource pool view is complete and is available for real-time querying by the scheduler on the router via shared memory or a remote dictionary service.

[0037] In addition, for newly connected edge devices, the router listens for device online events through a multicast heartbeat mechanism or a device registration center. Once a new device is detected, the automatic identification and resource registration process described in step S1 is triggered without manual configuration or system restart.

[0038] Step S2: In response to the router receiving the AI ​​model, the AI ​​model is decomposed into multiple independently executable subgraphs based on computation graph analysis technology, and a computation cost model for each subgraph is established; combined with the hardware specification information of each AI acceleration hardware in the heterogeneous computing power resource pool view, the optimal subgraph-hardware mapping scheme that satisfies the latency constraint is searched, and a cross-device collaborative execution plan is generated. The cross-device collaborative execution plan includes subgraph partitioning, device mapping, data dependency, and synchronization point information. In step S2, the router first parses the received AI model. The AI ​​model is typically sent to the router in a serialized format (e.g., .pt, .pb, .onnx files) of a deep learning framework (e.g., PyTorch, TensorFlow, ONNX). The model parser deployed on the router reads this file and deserializes it into a directed acyclic computation graph (DAG) in memory, where nodes are operators and edges represent tensor data flows.

[0039] Next, the computational graph analysis technique employs a clustering-based graph segmentation algorithm to segment the directed acyclic computational graph. Specifically, the algorithm calculates the tensor data volume (i.e., data size, in MB) on each edge of the graph and the computational coupling between adjacent operators. With the optimization objective of minimizing cross-subgraph data transfer overhead, highly coupled operators are aggregated into the same subgraph, and sparsely dependent boundaries are used as subgraph segmentation points. Each segmented subgraph contains a complete sequence of operators, and subgraphs interact with each other only through a few tensor interfaces, thus ensuring that each subgraph can be independently scheduled and executed.

[0040] Subsequently, for each subgraph obtained from the segmentation, the cost analyzer on the router calculates the total floating-point operations (FLOPs) based on the types and number of operators contained in the subgraph, and estimates the memory read / write traffic (in MB) by combining the input / output tensor dimensions of the subgraph. Combining these two metrics, a computational cost model is established with expected execution latency and energy consumption as outputs. This model can be represented as a mapping function with hardware type as a parameter, used for subsequent mapping decisions.

[0041] During the search for the optimal subgraph-hardware mapping scheme, the scheduler prioritizes minimizing the total end-to-end latency, constrained by the upper limits of computing power, memory capacity, and network bandwidth of each hardware component. It employs dynamic programming or reinforcement learning algorithms (such as Deep Q-Network) to search for policies in the global solution space. If reinforcement learning is used, the policy network is pre-trained in a simulation environment, allowing for rapid output of an approximate optimal mapping scheme during online inference with only forward propagation. The final generated cross-device collaborative execution plan comprises a directed task graph, where each task node is associated with a subgraph, a target device, and the specific acceleration hardware on that device. Edges are labeled with data dependencies and the locations of synchronization barriers.

[0042] Furthermore, as a preprocessing step in step S2, the router performs a series of hardware-aware graph optimizations on the computation graph before generating the execution plan. For example, consecutive convolutional layers, batch normalization layers, and activation function layers (ReLU or Sigmoid) are merged into a single operator node to reduce kernel startup overhead and memory read / write operations on intermediate tensors. Simultaneously, based on the target hardware's native precision preference, the operator precision format in the subgraph is adaptively converted between FP32, FP16, or INT8. If the target hardware supports INT8 fixed-point arithmetic and the precision loss is within acceptable limits, INT8 precision is preferentially used to obtain higher throughput and energy efficiency.

[0043] Step S3: In response to the AI ​​inference request, the AI ​​inference task corresponding to the AI ​​inference request is divided into multiple stages according to the cross-device collaborative execution plan, and mapped to the AI ​​acceleration hardware of different edge devices for pipelined parallel execution; during the execution process, the task arrival rate is monitored and the batch size is dynamically adjusted to aggregate multiple requests to form the optimal batch. In the specific implementation of step S3, after the camera acquires video frames, it first performs preprocessing operations such as decoding and size normalization on the raw image data to form a tensor that meets the model input requirements. The camera then sends an AI inference request to the router, which carries the task ID, the address of the input tensor in the shared memory pool, and the required quality of service level (such as maximum latency tolerance).

[0044] Upon receiving the request, the router scheduler instantiates an inference pipeline for the request based on the cross-device collaborative execution plan generated in step S2. This pipeline divides the complete inference task into multiple stages: preprocessing stage (e.g., image scaling, color space conversion, normalization), feature extraction stage (e.g., backbone network computation of deep convolutional neural networks), and postprocessing stage (e.g., bounding box regression, nonmaximum suppression, class probability decoding).

[0045] In pipelined parallel execution mode, different stages are mapped to AI acceleration hardware on different edge devices. A typical mapping is as follows: the camera's own lightweight NPU performs the preprocessing stage and shallow feature extraction to reduce raw data transmission; the gateway's mid-range GPU or NPU receives intermediate feature maps from multiple cameras under its jurisdiction and performs mid-level feature fusion and high-level semantic extraction; the router's high-performance NPU acts as the final aggregation node, performing global pooling of feature maps and computation of the detection head output.

[0046] To ensure continuous pipeline operation, an asynchronous execution model is adopted among the hardware nodes. After each node completes the computation of the current stage, it puts the output tensor into the sending queue and immediately begins processing the next batch of data without waiting for subsequent stages to complete, thereby maximizing hardware parallelism.

[0047] Meanwhile, the mechanism for dynamically adjusting batch size is implemented as follows: The system maintains a request queue in the router's memory to cache unscheduled inference requests. The scheduler continuously monitors the rate of change of the queue's depth, i.e., the task arrival rate. When the request queue depth exceeds a first threshold (e.g., 10 requests), the system is determined to be under high load. The scheduler concatenates the input tensors of multiple requests along the batch dimension into a large tensor, forming a large batch input, which is then sent to the hardware for execution all at once, thereby reducing the number of kernel calls and improving computing power utilization. When the request queue depth is below a second threshold (e.g., 2 requests), the system is determined to be under low load. The scheduler immediately processes a single request, avoiding additional latency caused by waiting to complete the batch.

[0048] For cross-device collaboration scenarios, the gateway device performs regional-level dynamic batch processing: the gateway collects feature maps generated by multiple cameras under its jurisdiction within a similar time window, aggregates them into a logical batch, and then sends them to the router for processing via a high-speed link. This reduces the number of uplink transmission packets and provides the router with greater batch processing granularity.

[0049] Step S4: During the parallel execution of the pipeline, tensor data is directly transferred between the AI ​​acceleration hardware of different edge devices through the zero-copy data channel established between each edge device, thereby bypassing the memory transfer of the central processing unit. In step S4, the establishment of the zero-copy data channel is completed during the system initialization phase or before the first data transmission. Taking a Remote Direct Memory Access (RDMA)-based implementation as an example, the network interface cards (NICs) of each edge device support RoCE (RDMA over Converged Ethernet) or the InfiniBand protocol. After the AI ​​acceleration hardware of the source device (e.g., the gateway) completes the computation, its output tensor is stored in a dedicated memory area of ​​that hardware (such as the on-chip SRAM of the NPU or the video memory of the GPU).

[0050] The source device's driver layer obtains the address and length of the tensor in physical memory and initiates an asynchronous write operation directly to the AI-accelerated hardware memory region of the destination device (e.g., a router) via the RDMA network card. The entire data transmission path is completely controlled by the network card and the high-speed interconnect bus, without passing through the CPU kernel / user space of the source or destination, thus achieving zero CPU intervention at the end.

[0051] During execution, each edge device pre-negotiates the tensor data format (such as row-major or column-major order) and memory alignment requirements through the control plane. Before initiating data transmission, the sending end notifies the receiving end of the imminent arrival of data via hardware semaphores or a doorbell mechanism. After the data is written, the AI ​​acceleration hardware at the receiving end senses that the data is ready through interrupts or polling, and then directly reads the tensor data in that memory region for the next stage of computation.

[0052] As an alternative implementation, for devices lacking RDMA capabilities, the zero-copy data channel can be implemented using shared memory combined with a Direct Memory Access (DMA) engine. For example, within the same gateway device, the NPU and GPU transfer data via a DMA controller connected through a PCIe bus, also without CPU intervention. For cross-device scenarios, an Ethernet-based DMA over TCP / IP offload engine is used, where the network card hardware handles packet encapsulation and parsing, further reducing transmission latency.

[0053] Step S5: Monitor the operating status of each AI acceleration hardware in real time, and perform dynamic voltage and frequency adjustment and task migration coordination operation according to the operating status; establish the power consumption-performance model of the AI ​​acceleration hardware to predict the risk of overheating or power failure; when hardware abnormality or performance degradation is detected, migrate the currently running AI inference task to the backup hardware or switch to the central processing unit for execution; when the hardware failure is recovered, automatically perform load rebalancing operation to migrate the AI ​​inference task back to the original hardware to restore system performance.

[0054] In step S5, the system equips each AI acceleration hardware with a status monitoring agent, which runs on the device or is centrally polled by the router. The monitoring agent reads hardware sensor data at millisecond sampling intervals, including: core temperature (°C), total board power consumption (W), and power supply mode (external AC power or battery).

[0055] Based on collected historical data, the system management module on the router constructs a power consumption-performance model for each hardware component. This model describes the relationship between power consumption and computational throughput when the hardware executes various operators at a specific operating voltage and frequency. Using this model, the system can predict when the thermal design power (TDP) limit will be reached or overheat protection will be triggered if the hardware temperature continues to rise under the current load intensity.

[0056] The specific strategy for the coordinated operation of Dynamic Voltage Frequency Scaling (DVFS) and task migration is as follows: When the system management module detects that the temperature of a certain AI acceleration hardware exceeds a first temperature threshold (e.g., 85°C) or its power consumption exceeds the long-term power consumption limit, it first issues a frequency adjustment command to the hardware, reducing its core frequency by one level (e.g., from 1.0GHz to 0.8GHz) to reduce instantaneous heat generation. Simultaneously, it assesses whether the hardware's processing capability for the currently executed task can still meet latency constraints after the frequency reduction; if not, it immediately initiates the task migration process, migrating the incomplete inference task context (including model weights and intermediate activation tensors) to other similar hardware or a general-purpose CPU with lower load through a zero-copy channel.

[0057] When an unrecoverable hardware failure is detected (such as consecutive memory check errors exceeding limits) or the power supply system switches to battery power, the system management module suspends all executing and queued inference tasks on that hardware and reallocates the tasks to backup acceleration hardware or a general-purpose CPU according to a predefined failover strategy. During CPU execution, software acceleration can be achieved using its vector instruction set (such as AVX-512), although slower, ensuring uninterrupted service.

[0058] When a hardware failure recovery signal (such as when a device comes back online) is triggered, the system management module performs a load rebalancing operation: comparing the current available computing power of the original hardware with the load of other devices, and gradually migrating back the previously migrated tasks in order of priority until the load ratio of each device is restored to the pre-failure level or reaches a new global optimal state.

[0059] For battery-powered camera devices, the system also monitors the remaining battery percentage. When the battery level falls below a set threshold (e.g., 20%), the system negotiates with the router to dynamically reduce the frequency of inference requests sent by the camera (e.g., from 30fps to 10fps) and requests the router to use a low-precision (INT8) model version for inference. This significantly reduces the encoding, transmission, and computational power consumption of the camera, extending the device's battery life.

[0060] In step S1, the unified driver interface shields the differences in instruction set architecture, memory model, operator type and precision format of AI acceleration hardware from different manufacturers, and provides a standardized computing power service interface to the upper layer; the hardware specification information includes computing power, memory capacity, supported operator types and precision formats.

[0061] In step S1, the construction of the heterogeneous computing power resource pool view supports the automatic discovery of newly connected devices: when a new edge device is connected to the system, its AI acceleration hardware is automatically identified and its hardware specification information is automatically registered to the resource pool.

[0062] In step S2, the computational cost model is established based on the number of floating-point operations and memory access patterns of the subgraph; the search for the optimal subgraph-hardware mapping scheme is implemented using dynamic programming or reinforcement learning algorithms.

[0063] Step S2 further includes: before generating the cross-device collaborative execution plan, performing operator fusion, precision conversion, and memory layout optimization operations for different AI acceleration hardware; wherein, the operator fusion includes merging convolutional layers, batch normalization layers, and activation function layers; the precision conversion includes adaptive selection of FP32, FP16, and INT8 precision formats.

[0064] In step S3, the multiple stages of the AI ​​inference task include a preprocessing stage, a feature extraction stage, and a post-processing stage; the pipeline parallel execution includes: the preprocessing stage and / or the feature extraction stage being executed by the neural network processing unit of the camera, the feature fusion operation in the feature extraction stage and / or the post-processing stage being executed by the neural network processing unit or the graphics processing unit of the gateway, and the detection output operation in the post-processing stage being executed by the high-performance neural network processing unit of the router. The dynamic batch size adjustment includes: monitoring the request queue used to cache the AI ​​inference requests; increasing the batch size when the request queue accumulates to improve hardware utilization; and decreasing the batch size when the request queue is cleared to reduce latency. The dynamic batch processing supports cross-device collaboration, whereby the gateway aggregates feature data from multiple cameras under its jurisdiction to form a large batch and sends it to the router for processing.

[0065] In step S4, the zero-copy data channel is implemented based on shared memory, remote direct memory access, or a direct memory access engine, and is used to directly exchange tensor data between AI acceleration hardware on different edge devices.

[0066] In step S5, the operating status includes temperature, power consumption and power supply status; the dynamic voltage and frequency adjustment and task migration coordinated operation includes: when under high load, the AI ​​acceleration hardware with the highest energy efficiency ratio is used first, and when under low load, it is switched to low power mode or the AI ​​inference task is migrated to backup hardware or central processing unit for execution. The hardware anomaly or performance degradation includes hardware overheating and frequency reduction, hardware failure, and switching from mains power failure to battery power; the method also includes: when the battery power of the battery-powered device is detected to be lower than the power threshold, dynamically adjusting the inference accuracy and inference frequency of the AI ​​inference task according to the remaining power.

[0067] The method further includes: Step S6: Periodically evaluate the actual execution efficiency and energy consumption ratio of each AI acceleration hardware, and adjust the subgraph partitioning boundary and device mapping strategy online; predict the future load based on the spatiotemporal distribution pattern of task arrival, and perform pre-migration operations for AI inference tasks based on the prediction results; The periodic evaluation period is 5 seconds; the online adjustment strategy for subgraph partitioning boundaries and device mapping includes: when it is detected that the actual execution efficiency of a certain operator on the current neural network processing unit is lower than the expected efficiency threshold, the subgraph corresponding to the operator is dynamically migrated to the graphics processing unit for execution.

[0068] During the closed-loop optimization process in step S6, the performance analyzer on the router counts the number of subgraph instances executed on each hardware device and the cumulative time consumed over a period of 5 seconds, calculating the actual throughput (FPS) and actual energy efficiency ratio (unit: FPS / W). These measured data are compared with the theoretical estimates calculated by the cost model in step S2.

[0069] When the system detects that the measured execution time of an operator (e.g., depthwise separable convolution) on the currently specified neural network processing unit consistently exceeds 1.2 times the theoretically estimated time, it determines that the mapping scheme is not optimal. In this case, the scheduler triggers an online reconfiguration: migrating the subgraph to which the operator belongs from the current NPU to a graphics processing unit on the same device or another device for execution, and updating the mapping relationships in the cross-device collaborative execution plan. After remapping, subsequent inference requests will be executed according to the new plan, while old requests currently being processed in the pipeline will continue to be completed according to the original plan, ensuring consistency in the execution process.

[0070] Simultaneously, based on historical task arrival records, the system uses time series analysis or Long Short-Term Memory (LSTM) networks to predict load intensity and distribution patterns for the next few to tens of seconds. If a sudden video stream is predicted to occur from a camera under a gateway in a certain area, the system pre-migrates some lightweight inference subgraphs on the router to that gateway for execution, or pre-allocates computing resources from the router's high-performance NPU to avoid latency spikes caused by scheduling and resource contention when requests arrive. These prediction and pre-migrating operations are executed cyclically in the background, forming an adaptive closed loop of "perception-analysis-decision-execution," continuously improving the system's long-term average performance in dynamic environments.

[0071] The router serves as the global computing power scheduling center and model optimization center, connecting to at least one gateway to form a star topology; the gateway serves as a regional computing power aggregation point, connecting to at least one subordinate camera to perform regional task scheduling and data aggregation; the camera serves as a data acquisition and front-end inference node, used to perform video stream acquisition and basic AI inference tasks. The router and the gateway are connected via 10 Gigabit Ethernet or fiber optic cable; the gateway and the camera are connected via Gigabit Ethernet or Power over Ethernet; the zero-copy data channel is established between each edge device through a high-speed interconnect channel, which includes PCIe, Ethernet and Wi-Fi 6.

[0072] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for accelerating and optimizing AI inference through router gateway-camera collaboration, characterized in that, Includes the following steps: Step S1: After the edge device starts up, it automatically detects the local AI acceleration hardware and collects hardware specification information through the unified driver interface, and reports it to the router, which serves as the global scheduling center. The edge device includes the router, which serves as the global scheduling center, at least one gateway, and at least one camera under each gateway. The router summarizes the hardware specification information reported by each device and constructs a heterogeneous computing power resource pool view containing the AI ​​acceleration hardware and its hardware specification information of each edge device. After the camera collects the video stream, it initiates an AI inference request. Step S2: In response to the router receiving the AI ​​model, the AI ​​model is decomposed into multiple independently executable subgraphs based on computation graph analysis technology, and a computation cost model for each subgraph is established; combined with the hardware specification information of each AI acceleration hardware in the heterogeneous computing power resource pool view, the optimal subgraph-hardware mapping scheme that satisfies the latency constraint is searched, and a cross-device collaborative execution plan is generated. The cross-device collaborative execution plan includes subgraph partitioning, device mapping, data dependency, and synchronization point information. Step S3: In response to the AI ​​inference request, the AI ​​inference task corresponding to the AI ​​inference request is divided into multiple stages according to the cross-device collaborative execution plan, and mapped to the AI ​​acceleration hardware of different edge devices for pipelined parallel execution; during the execution process, the task arrival rate is monitored and the batch size is dynamically adjusted to aggregate multiple requests to form the optimal batch. Step S4: During the parallel execution of the pipeline, tensor data is directly transferred between the AI ​​acceleration hardware of different edge devices through the zero-copy data channel established between each edge device, thereby bypassing the memory transfer of the central processing unit. Step S5: Monitor the operating status of each AI acceleration hardware in real time, and perform dynamic voltage and frequency adjustment and task migration coordination operation according to the operating status; establish the power consumption-performance model of the AI ​​acceleration hardware to predict the risk of overheating or power failure; when hardware abnormality or performance degradation is detected, migrate the currently running AI inference task to the backup hardware or switch to the central processing unit for execution. Once the hardware failure is resolved, a load rebalancing operation is automatically performed to migrate the AI ​​inference tasks back to the original hardware in order to restore system performance.

2. The router gateway camera collaborative AI inference acceleration optimization method as described in claim 1, characterized in that, In step S1, the unified driver interface shields the differences in instruction set architecture, memory model, operator type and precision format of AI acceleration hardware from different manufacturers, and provides a standardized computing power service interface to the upper layer; the hardware specification information includes computing power, memory capacity, supported operator types and precision formats.

3. The router gateway camera collaborative AI inference acceleration optimization method as described in claim 1, characterized in that, In step S1, the construction of the heterogeneous computing power resource pool view supports the automatic discovery of newly connected devices: when a new edge device is connected to the system, its AI acceleration hardware is automatically identified and its hardware specification information is automatically registered to the resource pool.

4. The router gateway camera collaborative AI inference acceleration optimization method as described in claim 1, characterized in that, In step S2, the computational cost model is established based on the number of floating-point operations and memory access patterns of the subgraph; the search for the optimal subgraph-hardware mapping scheme is implemented using dynamic programming or reinforcement learning algorithms.

5. The router gateway camera collaborative AI inference acceleration optimization method as described in claim 1, characterized in that, Step S2 further includes: before generating the cross-device collaborative execution plan, performing operator fusion, precision conversion, and memory layout optimization operations for different AI acceleration hardware; wherein, the operator fusion includes merging convolutional layers, batch normalization layers, and activation function layers; the precision conversion includes adaptive selection of FP32, FP16, and INT8 precision formats.

6. The router gateway camera collaborative AI inference acceleration optimization method as described in claim 1, characterized in that, In step S3, the multiple stages of the AI ​​inference task include a preprocessing stage, a feature extraction stage, and a post-processing stage; the pipeline parallel execution includes: the preprocessing stage and / or the feature extraction stage being executed by the neural network processing unit of the camera, the feature fusion operation in the feature extraction stage and / or the post-processing stage being executed by the neural network processing unit or the graphics processing unit of the gateway, and the detection output operation in the post-processing stage being executed by the high-performance neural network processing unit of the router. The dynamic batch size adjustment includes: monitoring the request queue used to cache the AI ​​inference requests; increasing the batch size when the request queue accumulates to improve hardware utilization; and decreasing the batch size when the request queue is cleared to reduce latency. The dynamic batch processing supports cross-device collaboration, whereby the gateway aggregates feature data from multiple cameras under its jurisdiction to form a large batch and sends it to the router for processing.

7. The router gateway camera collaborative AI inference acceleration optimization method as described in claim 1, characterized in that, In step S4, the zero-copy data channel is implemented based on shared memory, remote direct memory access, or a direct memory access engine, and is used to directly exchange tensor data between AI acceleration hardware on different edge devices.

8. The router gateway camera collaborative AI inference acceleration optimization method as described in claim 1, characterized in that, In step S5, the operating status includes temperature, power consumption and power supply status; the dynamic voltage and frequency adjustment and task migration coordinated operation includes: when under high load, the AI ​​acceleration hardware with the highest energy efficiency ratio is used first, and when under low load, it is switched to low power mode or the AI ​​inference task is migrated to backup hardware or central processing unit for execution. The hardware anomaly or performance degradation includes hardware overheating and frequency reduction, hardware failure, and switching from mains power failure to battery power; the method also includes: when the battery power of the battery-powered device is detected to be lower than the power threshold, dynamically adjusting the inference accuracy and inference frequency of the AI ​​inference task according to the remaining power.

9. The router gateway camera collaborative AI inference acceleration optimization method as described in claim 1, characterized in that, The method further includes: Step S6: Periodically evaluate the actual execution efficiency and energy consumption ratio of each AI acceleration hardware, and adjust the subgraph partitioning boundary and device mapping strategy online; predict the future load based on the spatiotemporal distribution pattern of task arrival, and perform pre-migration operations for AI inference tasks based on the prediction results; The periodic evaluation period is 5 seconds; the online adjustment strategy for subgraph partitioning boundaries and device mapping includes: when it is detected that the actual execution efficiency of a certain operator on the current neural network processing unit is lower than the expected efficiency threshold, the subgraph corresponding to the operator is dynamically migrated to the graphics processing unit for execution.

10. A router gateway camera collaborative AI inference acceleration optimization method as described in any one of claims 1 to 9, characterized in that, The router serves as the global computing power scheduling center and model optimization center, connecting to at least one gateway to form a star topology; the gateway serves as a regional computing power aggregation point, connecting to at least one subordinate camera to perform regional task scheduling and data aggregation; the camera serves as a data acquisition and front-end inference node, used to perform video stream acquisition and basic AI inference tasks. The router and the gateway are connected via 10 Gigabit Ethernet or fiber optic cable; the gateway and the camera are connected via Gigabit Ethernet or Power over Ethernet; the zero-copy data channel is established between each edge device through a high-speed interconnect channel, which includes PCIe, Ethernet and Wi-Fi 6.

Images