Real-time dynamic inference optimization method and system for ai gateway

Abstract

The application discloses a real-time dynamic inference optimization method and system for an AI gateway, and belongs to the technical field of artificial intelligence and edge computing. The technical problem to be solved by the application is how to optimize the real-time dynamic inference capability of the AI gateway and improve the real-time inference efficiency of the AI gateway under complex tasks. The technical scheme is as follows: model lightweight preprocessing: an AI model is constructed by using a lightweight network architecture, and the model calculation complexity and parameter quantity are reduced by using deep separable convolution and channel shuffling technology; and the constructed model is subjected to pruning, quantization and knowledge distillation model compression processing; real-time dynamic inference decision: in the AI gateway running process, the complexity of input data, the real-time requirement of the current task and the computing power resource use condition of the AI gateway are monitored in real time, and adaptive inference strategy decision is made according to the monitoring result; inference engine and compilation optimization; data preprocessing and edge intelligence; edge-cloud collaborative inference.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and edge computing technology, specifically a real-time dynamic inference optimization method and system for AI gateways. Background Technology

[0002] With the widespread application of artificial intelligence (AI) technology in edge computing scenarios, the importance of AI gateways is becoming increasingly prominent. However, current AI gateways face numerous challenges in real-time inference. On the one hand, AI models are complex and have numerous parameters. For example, traditional deep neural networks require a large number of floating-point operations for each inference iteration when performing tasks such as image recognition and object detection, placing extremely high demands on computing power. However, due to limitations in the size, power consumption, and heat dissipation of AI gateways, it is difficult to equip them with high-performance hardware, resulting in significant inference latency. On the other hand, different tasks have significantly different computing power requirements, and the complexity and real-time requirements of input data also vary. Traditional fixed resource allocation and static inference methods cannot adapt to these dynamic changes, easily leading to wasted computing power or task backlog, making it difficult to meet real-time requirements. Furthermore, most existing inference optimization techniques focus on a single dimension, such as compressing the model or optimizing only the inference engine, making it difficult to achieve efficient end-to-end real-time inference.

[0003] Therefore, how to optimize the real-time dynamic reasoning capability of AI gateways and improve their real-time reasoning efficiency under complex tasks is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] The technical objective of this invention is to provide a real-time dynamic inference optimization method and system for AI gateways, in order to solve the problem of how to optimize the real-time dynamic inference capability of AI gateways and improve the real-time inference efficiency of AI gateways under complex tasks.

[0005] The technical objective of this invention is achieved as follows: a real-time dynamic inference optimization method for AI gateways, the specific method of which is as follows:

[0006] Lightweight model preprocessing: A lightweight network architecture is used to build the AI ​​model, and depthwise separable convolution and channel shuffling techniques are used to reduce the computational complexity and number of parameters of the model; and the built model is compressed by pruning, quantization and knowledge distillation.

[0007] Real-time dynamic reasoning and decision-making: During the operation of the AI ​​gateway, the complexity of the input data, the real-time requirements of the current task, and the usage of the AI ​​gateway's computing resources are monitored in real time. Based on the monitoring results, adaptive reasoning strategy decisions are made, and real-time resource scheduling is implemented based on task priority and computing resources.

[0008] Inference Engine and Compilation Optimization: Based on the hardware platform of the AI ​​gateway, a lightweight inference engine adapted to the hardware is selected, and the model computation graph is compiled and optimized. Continuous operators are merged, and the continuous operations of convolution, batch normalization, and activation functions are integrated into a single operator to reduce the number of memory accesses. Then, heterogeneous computing technology is used to reasonably allocate computing tasks to different hardware units of CPU, GPU, and NPU according to task type, and efficient task splitting and collaborative processing are achieved through OpenCL / OpenVX interface.

[0009] Data preprocessing and edge intelligence: The data received by the AI ​​gateway is preprocessed by denoising and feature dimensionality reduction, and a local data caching mechanism is established to store high-frequency input sample features. At the same time, an incremental learning algorithm is adopted so that when the local data distribution changes, only the affected model parameters are updated, avoiding retraining the entire model.

[0010] Edge-Cloud Collaborative Inference: Construct a hierarchical inference architecture, assign simple tasks with real-time requirements higher than a set threshold and computational load lower than a set threshold to the AI ​​gateway for processing, upload complex model training and global optimization tasks to the cloud, and use a combination of federated learning and model distillation to achieve data interaction and model updates between the edge and the cloud.

[0011] As preferred options, lightweight network architectures include the MobileNet series, ShuffleNet, and YOLO-Lite;

[0012] Pruning refers to the pruning operation performed through L1 / L2 regularization to remove redundant connections and reduce parameters in the model.

[0013] Quantization refers to converting model parameters from high-bit (e.g., 32-bit) to low-bit (e.g., 8-bit, 4-bit, or even binarization) to reduce memory usage and computational load.

[0014] Knowledge distillation refers to using teacher models to guide student model learning, thereby reducing the model size while ensuring model accuracy.

[0015] As a preferred option, the utilization of computing resources includes the utilization rate of CPU, GPU, and NPU, memory usage, and queue length;

[0016] The adaptive inference strategy decision-making content is as follows: if the simplicity of the input data is lower than the set threshold and the real-time requirement is higher than the set threshold, then a lightweight model is used for inference first; if the simplicity of the input data is higher than the set threshold, then a high-precision model is used when computing power is sufficient, and the model is automatically downgraded when computing power is insufficient; for models with multi-branch structures, an early stopping mechanism is adopted to output results in advance, and results are output in advance when simple samples meet specific conditions to reduce unnecessary calculations.

[0017] Real-time resource scheduling based on task priority and computing resources involves prioritizing different tasks. In industrial scenarios, fault warning tasks are given higher priority than regular inspection tasks, and computing resources are allocated to them first. Model operating parameters are dynamically adjusted based on computing power usage. These parameters include inference frequency and data processing batch size.

[0018] As a preferred option, a lightweight inference engine adapted to the hardware includes deploying TensorRT on the NVIDIA GPU platform, leveraging TensorRT's support for FP16 / INT8 quantization and graph optimization features to accelerate model inference;

[0019] or,

[0020] Leverage the cross-platform and lightweight advantages of using ONNX Runtime or TFLite on ARM architecture devices.

[0021] As a preferred approach, the data preprocessing uses a spatiotemporal filtering algorithm to denoise the data collected by industrial sensor equipment, and employs PCA and LDA methods to perform feature dimensionality reduction on high-dimensional data, thereby reducing invalid data and computational load.

[0022] More preferably, the local data caching mechanism works as follows: when new data is input, it is first compared with cached features to quickly determine whether a conclusion can be drawn directly, reducing the number of full inferences performed by the model.

[0023] A real-time dynamic inference optimization system for AI gateways, the system comprising:

[0024] The model preprocessing module is used to build AI models using a lightweight network architecture, and to reduce the computational complexity and number of parameters of the models by using depthwise separable convolution and channel shuffling techniques; and to perform model compression processing such as pruning, quantization and knowledge distillation on the built models.

[0025] The dynamic decision-making module is used to monitor the complexity of input data, the real-time requirements of the current task, and the computing resources of the AI ​​gateway in real time during the operation of the AI ​​gateway. Based on the monitoring results, it makes adaptive reasoning strategy decisions and then implements real-time resource scheduling based on task priority and computing resources.

[0026] The inference optimization module is used to select a lightweight inference engine that is compatible with the hardware platform of the AI ​​gateway, and to compile and optimize the model computation graph. It merges continuous operators, combining the continuous operations of convolution, batch normalization and activation functions into a single operator to reduce the number of memory accesses. Then, it uses heterogeneous computing technology to reasonably allocate computing tasks to different hardware units of CPU, GPU and NPU according to task type, and realizes efficient task splitting and collaborative processing through OpenCL / OpenVX interface.

[0027] The data processing module is used to preprocess the data received by the AI ​​gateway by denoising and feature dimensionality reduction, and to establish a local data caching mechanism to store high-frequency input sample features. At the same time, it adopts an incremental learning algorithm so that when the local data distribution changes, only the affected model parameters are updated to avoid retraining the entire model.

[0028] The collaborative reasoning module is used to build a hierarchical reasoning architecture and adopts a combination of federated learning and model distillation to achieve task allocation and data interaction between the edge and the cloud.

[0029] As preferred options, lightweight network architectures include the MobileNet series, ShuffleNet, and YOLO-Lite;

[0030] Pruning refers to the pruning operation performed through L1 / L2 regularization to remove redundant connections and reduce parameters in the model.

[0031] Quantization refers to converting model parameters from high-bit (e.g., 32-bit) to low-bit (e.g., 8-bit, 4-bit, or even binarization) to reduce memory usage and computational load.

[0032] Knowledge distillation refers to using teacher models to guide student model learning, thereby reducing the model size while ensuring model accuracy.

[0033] As a preferred option, the utilization of computing resources includes the utilization rate of CPU, GPU, and NPU, memory usage, and queue length;

[0034] The adaptive inference strategy decision-making content is as follows: if the simplicity of the input data is lower than the set threshold and the real-time requirement is higher than the set threshold, then a lightweight model is used for inference first; if the simplicity of the input data is higher than the set threshold, then a high-precision model is used when computing power is sufficient, and the model is automatically downgraded when computing power is insufficient; for models with multi-branch structures, an early stopping mechanism is adopted to output results in advance, and results are output in advance when simple samples meet specific conditions to reduce unnecessary calculations.

[0035] Real-time resource scheduling based on task priority and computing resources involves prioritizing different tasks. In industrial scenarios, fault warning tasks are given higher priority than regular inspection tasks, and computing resources are allocated to them first. Model operating parameters are dynamically adjusted based on computing power usage. These parameters include inference frequency and data processing batch size.

[0036] Even better, a hardware-adaptive lightweight inference engine includes deploying TensorRT on the NVIDIA GPU platform, leveraging TensorRT's support for FP16 / INT8 quantization and graph optimization capabilities to accelerate model inference.

[0037] or,

[0038] Leverage the cross-platform and lightweight advantages of using ONNX Runtime or TFLite on ARM architecture devices;

[0039] Data preprocessing employs a spatiotemporal filtering algorithm to denoise the data collected by industrial sensor equipment, and utilizes PCA and LDA methods to perform feature dimensionality reduction on high-dimensional data, thereby reducing invalid data and computational load.

[0040] The local data caching mechanism works as follows: when new data is input, it is first compared with cached features to quickly determine whether a conclusion can be drawn directly, thus reducing the number of full inferences performed by the model.

[0041] The real-time dynamic inference optimization method and system for AI gateways of the present invention have the following advantages:

[0042] (I) This invention reduces model complexity through lightweight model preprocessing, employing a lightweight architecture, pruning, quantization, and knowledge distillation; real-time dynamic inference decision-making, dynamically adjusting strategies and scheduling resources based on data, tasks, and computing power; inference engine and compilation optimization, adapting to hardware, selecting engines, optimizing computation graphs, and utilizing heterogeneous computing; data preprocessing and edge intelligence, denoising and reducing dimensionality of data, establishing a caching mechanism, and employing incremental learning; edge-cloud collaborative inference, constructing a layered architecture, combining federated learning and model distillation to achieve task allocation and model updates, realizing multi-dimensional comprehensive optimization, significantly improving AI gateway inference efficiency, reducing latency, increasing resource utilization, reducing data transmission volume, and enhancing system intelligence and adaptability;

[0043] (II) This invention focuses on real-time dynamic inference optimization technology for AI gateways. As a key node connecting edge devices and the cloud, the AI ​​gateway undertakes the important task of real-time data processing and inference. This invention is committed to overcoming the problem of low real-time inference efficiency of AI gateways when processing complex tasks, providing technical support for improving the performance of artificial intelligence applications in edge computing scenarios, and improving the real-time inference efficiency of AI gateways under complex tasks.

[0044] (III) This invention implements optimization strategies from multiple dimensions, including model structure, inference decision, inference engine, data processing and edge-cloud collaboration, to enable AI gateways to perform efficient real-time inference in different tasks and data scenarios, thereby improving the utilization of computing resources, reducing inference latency, meeting the real-time requirements of artificial intelligence applications in edge computing scenarios, and providing strong support for the widespread application of edge intelligence.

[0045] (iv) This invention adopts a multi-dimensional optimization strategy, which comprehensively optimizes the model structure, inference decision, inference engine, data processing and edge-cloud collaboration. Compared with the traditional single optimization method, it can significantly improve the inference efficiency of AI gateway and greatly reduce inference latency. For example, in industrial quality inspection scenarios, it can reduce the defect detection latency from 200ms to less than 50ms, which fully meets the real-time requirements.

[0046] (V) This invention realizes adaptive reasoning and dynamic resource scheduling of AI gateway under different tasks and data scenarios, improves the utilization rate of computing resources, effectively avoids computing waste and task accumulation, and can still ensure real-time processing of critical tasks under limited computing power.

[0047] (vi) The edge-cloud collaborative reasoning mechanism of the present invention reduces the amount of data transmission and the dependence on network bandwidth. At the same time, it achieves efficient model updates and optimization through federated learning and model distillation, thereby improving the intelligence level and adaptability of the entire system. Attached Figure Description

[0048] The invention will be further described below with reference to the accompanying drawings.

[0049] Appendix Figure 1 A flowchart illustrating a real-time dynamic inference optimization method for AI gateways;

[0050] Appendix Figure 2 This is a schematic diagram of the structure of a real-time dynamic inference optimization system for AI gateways. Detailed Implementation

[0051] The real-time dynamic inference optimization method and system for AI gateways of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1:

[0053] As attached Figure 1 As shown in the figure, this embodiment provides a real-time dynamic inference optimization method for AI gateways, which is as follows:

[0054] S1. Lightweight Model Preprocessing: A lightweight network architecture is used to build the AI ​​model. Depthwise separable convolution and channel shuffling techniques are used to reduce the computational complexity and number of parameters of the model. The constructed model is then subjected to pruning, quantization, and knowledge distillation for model compression.

[0055] S2. Real-time dynamic reasoning and decision-making: During the operation of the AI ​​gateway, the complexity of the input data, the real-time requirements of the current task, and the usage of the AI ​​gateway's computing resources are monitored in real time. Based on the monitoring results, adaptive reasoning strategy decisions are made, and real-time resource scheduling is implemented based on task priority and computing resources.

[0056] S3. Inference Engine and Compilation Optimization: Based on the hardware platform of the AI ​​gateway, a lightweight inference engine adapted to the hardware is selected, and the model computation graph is compiled and optimized. Continuous operators are merged, and the continuous operations of convolution, batch normalization, and activation functions are integrated into a single operator to reduce the number of memory accesses. Then, heterogeneous computing technology is used to reasonably allocate computing tasks to different hardware units of CPU, GPU, and NPU according to task type, and efficient task splitting and collaborative processing are achieved through OpenCL / OpenVX interface.

[0057] S4. Data Preprocessing and Edge Intelligence: The data received by the AI ​​gateway is preprocessed by denoising and feature dimensionality reduction, and a local data caching mechanism is established to store high-frequency input sample features. At the same time, an incremental learning algorithm is adopted so that when the local data distribution changes, only the affected model parameters are updated to avoid retraining the entire model.

[0058] S5. Edge-Cloud Collaborative Inference: Construct a hierarchical inference architecture, assign simple tasks with real-time requirements higher than a set threshold and computational load lower than a set threshold to the AI ​​gateway for processing, upload complex model training and global optimization tasks to the cloud, and use a combination of federated learning and model distillation to achieve data interaction and model updates between the edge and the cloud.

[0059] The lightweight network architecture in step S1 of this embodiment includes MobileNet series, ShuffleNet, and YOLO-Lite.

[0060] In this embodiment, pruning in step S1 refers to performing pruning operations through L1 / L2 regularization to remove redundant connections in the model and reduce parameters.

[0061] In step S1 of this embodiment, quantization refers to converting model parameters from high bits (such as 32 bits) to low bits (such as 8 bits, 4 bits, or even binarization) to reduce memory usage and computational load.

[0062] In step S1 of this embodiment, knowledge distillation refers to using a teacher model to guide student model learning, thereby reducing the model size while ensuring model accuracy.

[0063] In this embodiment, step S1, the lightweight model preprocessing example uses an image recognition task as an example, selecting MobileNetv3 as the basic network architecture to build the AI ​​model. The MobileNetv3 network structure is built using a deep learning framework (such as TensorFlow or PyTorch). This network, with its depthwise separable convolutions and SE modules (Squeeze-and-Excitation), significantly reduces computational cost while maintaining accuracy. Subsequently, the model is pruned, and L1 regularization parameters are set to gradually remove unimportant connections during training, reducing model parameters by approximately 40%. Next, 8-bit quantization is performed, converting the model parameters from 32-bit floating-point numbers to 8-bit integers. Testing on the ImageNet dataset shows that after quantization, the model's inference speed is increased by approximately 3 times, memory usage is reduced by 75%, and accuracy loss is only about 2%. Finally, knowledge distillation is applied, using ResNet-50 as the teacher model and MobileNetv3 as the student model. The teacher model's output guides the student model's learning, further improving the student model's accuracy.

[0064] The computing resource usage in step S2 of this embodiment includes the utilization rate of CPU, GPU, and NPU, memory usage rate, and queue length.

[0065] The adaptive inference strategy decision in step S2 of this embodiment is as follows: if the simplicity of the input data is lower than a set threshold and the real-time requirement is higher than a set threshold, then a lightweight model is used for inference first; if the simplicity of the input data is higher than a set threshold, then a high-precision model is used when computing power is sufficient, and the model is automatically downgraded when computing power is insufficient; for models with multi-branch structures, an early stopping mechanism is adopted to output results in advance, and results are output in advance when simple samples meet specific conditions, reducing unnecessary calculations.

[0066] In this embodiment, step S2, which implements real-time resource scheduling based on task priority and computing resources, specifically involves: prioritizing different tasks; in an industrial scenario, prioritizing fault warning tasks over regular inspection tasks and allocating computing resources accordingly; and dynamically adjusting model operating parameters based on computing power usage; wherein, model operating parameters include inference frequency and data processing batch size.

[0067] In the real-time dynamic inference decision-making embodiment of step S2 of this example, when the AI ​​gateway is running, the real-time monitoring module collects information such as CPU, GPU, NPU utilization, memory usage, and task queue length every 100ms. When the AI ​​gateway receives video stream data to perform target detection tasks, it first analyzes the complexity of the video frames: if there are few targets in the frame and the changes are gradual, it is determined to be simple data, and the current computing power is sufficient (e.g., CPU utilization is below 60%, NPU is idle), then the lightweight YOLO-Lite model is selected for inference; if there are many targets in the frame and they are moving quickly, it is determined to be complex data, and if the NPU utilization reaches 90% at this time, the model is automatically downgraded to the YOLO-Lite-Tiny model for inference. For multi-branch structure models, when the confidence of the output result of a sample in an early branch reaches a preset threshold (e.g., 0.9) during the inference process, the result is output in advance, and subsequent branch calculations are not performed. In terms of task priority, when equipment fault warning tasks and routine inspection tasks occur simultaneously in industrial scenarios, the fault warning task is set as the highest priority, and NPU resources are allocated to it first to ensure timely fault detection and response.

[0068] The lightweight inference engine for hardware adaptation in step S3 of this embodiment includes deploying TensorRT on the NVIDIA GPU platform, using TensorRT’s support for FP16 / INT8 quantization and graph optimization features to accelerate model inference.

[0069] or,

[0070] Leverage the cross-platform and lightweight advantages of using ONNX Runtime or TFLite on ARM architecture devices.

[0071] In step S3 of this embodiment, regarding the inference engine and compilation optimization, if the AI ​​gateway uses an NVIDIA Jetson series hardware platform, the TensorRT inference engine is deployed. The trained and quantized model is converted to a TensorRT-supported format. Utilizing its graph optimization capabilities, consecutive convolution, batch normalization, and activation function operations are merged, reducing memory access frequency and lowering memory access latency by approximately 60%. Simultaneously, TensorRT's support for INT8 quantization accelerates model inference, achieving approximately 4 times faster inference speed compared to native PyTorch models in the same object detection task. Regarding heterogeneous computing, the model feature extraction part is handled by the NPU, while control flow tasks such as target bounding box post-processing are assigned to the CPU. Efficient task splitting and collaboration are achieved through the OpenCL interface, reducing overall inference latency by approximately 30%.

[0072] In step S4 of this embodiment, the data preprocessing uses a spatiotemporal filtering algorithm to denoise the data collected by industrial sensor equipment, and uses PCA and LDA methods to perform feature dimensionality reduction on high-dimensional data to reduce invalid data and computational load.

[0073] The local data caching mechanism in step S4 of this embodiment is as follows: when new data is input, it is first compared with cached features to quickly determine whether a conclusion can be drawn directly, thereby reducing the number of full inferences by the model.

[0074] In step S4 of this embodiment, the data preprocessing and edge intelligence implementation involves the AI ​​gateway receiving industrial sensor data. First, it uses a Kalman filter algorithm for spatiotemporal filtering and noise reduction, reducing the data volume by approximately 50%. For high-dimensional sensor data (such as 1000-dimensional vibration data features), the PCA algorithm is used to reduce the dimensionality to 50 dimensions, reducing computation by approximately 95%. A local data cache is established to store sensor data features from the past week's normal operating conditions. When new data is input, it is compared with the cached features; if the similarity is higher than a threshold (e.g., 0.8), the device is directly determined to be normal, without requiring full inference. When new equipment is introduced to the factory, causing changes in data distribution, the iCaRL incremental learning algorithm is used to update only the affected model parameters. In task switching scenarios similar to the ImageNet dataset, the accuracy loss is controlled within 5%, and the entire model does not need to be retrained, saving significant computational resources and time.

[0075] In step S5 of this embodiment, the edge-cloud collaborative inference implementation constructs a layered inference architecture in a smart city video surveillance scenario. The AI ​​gateway is responsible for the initial processing of the real-time video stream, filtering out abnormal video frames and uploading only those frames and feature information that may contain anomalies (such as crowds or vehicle violations) to the cloud. The cloud handles complex model training and global analysis tasks, such as updating the abnormal behavior detection model weekly based on data uploaded from multiple AI gateways. Model updates employ a combination of federated learning and model distillation. After each AI gateway trains its model locally, it only uploads gradient information or distilled knowledge (such as feature mapping relationships) to the cloud, reducing communication volume by approximately 99% compared to uploading the original data. After cloud integration and optimization, the updated model is distributed to each AI gateway, achieving efficient model updates and optimization.

[0076] Example 2:

[0077] As attached Figure 2 As shown, this embodiment provides a real-time dynamic inference optimization system for AI gateways, the system comprising:

[0078] The model preprocessing module is used to build AI models using a lightweight network architecture, and to reduce the computational complexity and number of parameters of the models by using depthwise separable convolution and channel shuffling techniques; and to perform model compression processing such as pruning, quantization and knowledge distillation on the built models.

[0079] The dynamic decision-making module is used to monitor the complexity of input data, the real-time requirements of the current task, and the computing resources of the AI ​​gateway in real time during the operation of the AI ​​gateway. Based on the monitoring results, it makes adaptive reasoning strategy decisions and then implements real-time resource scheduling based on task priority and computing resources.

[0080] The inference optimization module is used to select a lightweight inference engine that is compatible with the hardware platform of the AI ​​gateway, and to compile and optimize the model computation graph. It merges continuous operators, combining the continuous operations of convolution, batch normalization and activation functions into a single operator to reduce the number of memory accesses. Then, it uses heterogeneous computing technology to reasonably allocate computing tasks to different hardware units of CPU, GPU and NPU according to task type, and realizes efficient task splitting and collaborative processing through OpenCL / OpenVX interface.

[0081] The data processing module is used to preprocess the data received by the AI ​​gateway by denoising and feature dimensionality reduction, and to establish a local data caching mechanism to store high-frequency input sample features. At the same time, it adopts an incremental learning algorithm so that when the local data distribution changes, only the affected model parameters are updated to avoid retraining the entire model.

[0082] The collaborative reasoning module is used to build a hierarchical reasoning architecture and adopts a combination of federated learning and model distillation to achieve task allocation and data interaction between the edge and the cloud.

[0083] The lightweight network architectures in this embodiment include MobileNet series, ShuffleNet, and YOLO-Lite.

[0084] In this embodiment, pruning refers to performing pruning operations through L1 / L2 regularization to remove redundant connections in the model and reduce parameters.

[0085] In this embodiment, quantization refers to converting model parameters from high bits (such as 32 bits) to low bits (such as 8 bits, 4 bits, or even binarization) to reduce memory usage and computational load.

[0086] In this embodiment, knowledge distillation refers to using a teacher model to guide student model learning, thereby reducing the model size while ensuring model accuracy.

[0087] The specific working process of the model preprocessing module in this embodiment is as follows:

[0088] ① Use lightweight network architectures designed specifically for edge computing scenarios, such as the MobileNet series, ShuffleNet, and YOLO-Lite, to build AI models. Employ techniques such as depthwise separable convolution and channel shuffling to significantly reduce the computational complexity and number of parameters of the model.

[0089] ② Compress the constructed model: perform pruning operations through L1 / L2 regularization to remove redundant connections and reduce parameters; perform quantization to convert model parameters from high bits (e.g., 32 bits) to low bits (e.g., 8 bits, 4 bits, or even binarization) to reduce memory usage and computation; and use knowledge distillation technology to guide student model learning with the help of teacher models to reduce the model size while ensuring model accuracy.

[0090] The computing resource usage in this embodiment includes the utilization rate of CPU, GPU, and NPU, memory usage rate, and queue length.

[0091] The adaptive inference strategy decision-making content in this embodiment is as follows: if the simplicity of the input data is lower than a set threshold and the real-time requirement is higher than a set threshold, then a lightweight model is used for inference first; if the simplicity of the input data is higher than a set threshold, then a high-precision model is used when computing power is sufficient, and the model is automatically downgraded when computing power is insufficient; for models with multi-branch structures, an early stopping mechanism is adopted to output results in advance, and when simple samples meet specific conditions, the results are output in advance to reduce unnecessary calculations.

[0092] In this embodiment, the real-time resource scheduling based on task priority and computing resources is implemented as follows: different tasks are prioritized. In an industrial scenario, the priority of fault early warning tasks is set higher than that of regular inspection tasks, and computing resources are allocated to them first. The model running parameters are dynamically adjusted according to the computing power usage. The model running parameters include inference frequency and data processing batch size.

[0093] The working process of the dynamic decision-making module in this embodiment is as follows:

[0094] ① During the operation of the AI ​​gateway, the complexity of input data, the real-time requirements of the current task, and the usage of AI gateway computing resources are monitored in real time, covering indicators such as CPU, GPU, NPU utilization, memory usage, and queue length.

[0095] ② Make adaptive inference strategy decisions based on monitoring results: If the input data is simple and real-time requirements are high, lightweight model inference is preferred; if the input data is complex, high-precision model is used when computing power is sufficient, and the model is automatically downgraded when computing power is insufficient; for multi-branch structure models, an early stopping mechanism is used to output results in advance when simple samples meet specific conditions, reducing unnecessary calculations.

[0096] ③ Implement real-time resource scheduling based on task priority and computing resources: prioritize different tasks. For example, in industrial scenarios, prioritize fault warning tasks over regular inspection tasks and allocate computing resources to them first. Dynamically adjust model running parameters, such as inference frequency and data processing batch size, according to computing power usage.

[0097] The lightweight inference engine adapted to the hardware in this embodiment includes deploying TensorRT on the NVIDIA GPU platform, which accelerates model inference by leveraging TensorRT's support for FP16 / INT8 quantization and graph optimization features.

[0098] or,

[0099] Leverage the cross-platform and lightweight advantages of using ONNX Runtime or TFLite on ARM architecture devices.

[0100] The working process of the inference optimization module in this embodiment is as follows:

[0101] ① Adapt lightweight inference engines according to the AI ​​gateway hardware platform: Deploy TensorRT on the NVIDIA GPU platform to accelerate model inference by leveraging its support for FP16 / INT8 quantization and graph optimization functions; Use ONNXRuntime or TFLite on ARM architecture devices to take advantage of their cross-platform and lightweight features.

[0102] ② Compile and optimize the model computation graph: merge continuous operators, integrate continuous operations such as convolution, batch normalization, and activation functions into a single operator to reduce memory access frequency; utilize heterogeneous computing technology to rationally allocate computational tasks to hardware units such as CPU, GPU, and NPU according to task type, and achieve efficient task splitting and collaborative processing through interfaces such as OpenCL / OpenVX.

[0103] In this embodiment, the data preprocessing uses a spatiotemporal filtering algorithm to denoise the data collected by industrial sensor equipment, and uses PCA and LDA methods to perform feature dimensionality reduction on high-dimensional data to reduce invalid data and computational load.

[0104] The local data caching mechanism in this embodiment is as follows: when new data is input, it is first compared with cached features to quickly determine whether a conclusion can be drawn directly, thereby reducing the number of full inferences by the model.

[0105] The data processing module in this embodiment works as follows:

[0106] ① After receiving the data, the AI ​​gateway first performs data preprocessing: it uses spatiotemporal filtering algorithms to denoise the data collected by industrial sensors and other devices, and uses methods such as PCA and LDA to perform feature dimensionality reduction on high-dimensional data to reduce invalid data and computational load.

[0107] ② Establish a local data caching mechanism to store features of high-frequency input samples. When new data is input, it is first compared with the cached features to quickly determine whether a conclusion can be drawn directly, reducing the number of full inferences by the model; at the same time, an incremental learning algorithm is adopted so that when the local data distribution changes, only the affected model parameters are updated, avoiding retraining the entire model.

[0108] The specific working process of the collaborative reasoning module in this implementation example is as follows:

[0109] ① Construct a hierarchical inference architecture: Assign simple tasks with high real-time requirements and low computational load to the AI ​​gateway for processing, such as video frame filtering and simple image recognition; upload complex model training and global optimization tasks to the cloud, such as weekly model updates and iterations.

[0110] ② A combination of federated learning and model distillation is adopted: Each AI gateway trains its model locally using local data and only uploads the gradient information or distilled knowledge obtained from the training to the cloud, reducing the amount of data transmission; after the cloud integrates and optimizes the information uploaded by each gateway, it distributes the updated model to the AI ​​gateway.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time dynamic inference optimization method for AI gateways, characterized in that, The method is as follows: Lightweight model preprocessing: A lightweight network architecture is used to build the AI ​​model. Depthwise separable convolution and channel shuffling techniques are used to reduce the computational complexity and number of parameters of the model. The constructed model is then subjected to pruning, quantization and knowledge distillation for model compression. Real-time dynamic reasoning and decision-making: During the operation of the AI ​​gateway, the complexity of the input data, the real-time requirements of the current task, and the computing resources of the AI ​​gateway are monitored in real time. Based on the monitoring results, adaptive reasoning strategy decisions are made, and real-time resource scheduling is implemented based on task priority and computing resources. Inference Engine and Compilation Optimization: Based on the hardware platform of the AI ​​gateway, a lightweight inference engine adapted to the hardware is selected, and the model computation graph is compiled and optimized. Continuous operators are merged, and the continuous operations of convolution, batch normalization, and activation functions are integrated into a single operator to reduce the number of memory accesses. Then, heterogeneous computing technology is used to reasonably allocate computing tasks to different hardware units of CPU, GPU, and NPU according to task type, and efficient task splitting and collaborative processing are achieved through OpenCL / OpenVX interface. Data preprocessing and edge intelligence: The data received by the AI ​​gateway is preprocessed by denoising and feature dimensionality reduction, and a local data caching mechanism is established to store high-frequency input sample features. At the same time, an incremental learning algorithm is adopted so that when the local data distribution changes, only the affected model parameters are updated. Edge-Cloud Collaborative Inference: Construct a hierarchical inference architecture, assign simple tasks with real-time requirements higher than a set threshold and computational load lower than a set threshold to the AI ​​gateway for processing, upload complex model training and global optimization tasks to the cloud, and use a combination of federated learning and model distillation to achieve data interaction and model updates between the edge and the cloud.

2. The real-time dynamic inference optimization method for AI gateways according to claim 1, characterized in that, Lightweight network architectures include the MobileNet series, ShuffleNet, and YOLO-Lite; Pruning refers to the pruning operation performed through L1 / L2 regularization to remove redundant connections and reduce parameters in the model. Quantization refers to converting model parameters from high-bit to low-bit, thereby reducing memory usage and computational load. Knowledge distillation refers to using teacher models to guide student model learning, thereby reducing the model size while ensuring model accuracy.

3. The real-time dynamic inference optimization method for AI gateways according to claim 1, characterized in that, The utilization of computing resources includes the utilization rate of CPU, GPU, and NPU, memory usage, and queue length; The adaptive inference strategy decision-making content is as follows: if the simplicity of the input data is lower than the set threshold and the real-time requirement is higher than the set threshold, then a lightweight model is used for inference first; if the simplicity of the input data is higher than the set threshold, then a high-precision model is used when computing power is sufficient, and the model is automatically downgraded when computing power is insufficient; for models with multi-branch structures, an early stopping mechanism is adopted to output results in advance, and results are output in advance when simple samples meet specific conditions to reduce unnecessary calculations. Real-time resource scheduling based on task priority and computing resources involves prioritizing different tasks. In industrial scenarios, fault warning tasks are given higher priority than regular inspection tasks, and computing resources are allocated to them first. Model operating parameters are dynamically adjusted based on computing power usage. These parameters include inference frequency and data processing batch size.

4. The real-time dynamic inference optimization method for AI gateways according to claim 1, characterized in that, The hardware-adaptive lightweight inference engine includes deploying TensorRT on the NVIDIA GPU platform, leveraging TensorRT's support for FP16 / INT8 quantization and graph optimization features to accelerate model inference; or, Leverage the cross-platform and lightweight advantages of using ONNX Runtime or TFLite on ARM architecture devices.

5. The real-time dynamic inference optimization method for AI gateways according to claim 1, characterized in that, Data preprocessing employs a spatiotemporal filtering algorithm to denoise the data collected by industrial sensor equipment, and utilizes PCA and LDA methods to perform feature dimensionality reduction on high-dimensional data, thereby reducing invalid data and computational load.

6. The real-time dynamic inference optimization method for AI gateways according to any one of claims 1 to 5, characterized in that, The local data caching mechanism works as follows: when new data is input, it is first compared with cached features to quickly determine whether a conclusion can be drawn directly, thus reducing the number of full inferences performed by the model.

7. A real-time dynamic inference optimization system for AI gateways, characterized in that, The system includes: The model preprocessing module is used to build AI models using a lightweight network architecture, and to reduce the computational complexity and number of parameters of the models by using depthwise separable convolution and channel shuffling techniques; and to perform model compression processing such as pruning, quantization and knowledge distillation on the built models. The dynamic decision-making module is used to monitor the complexity of input data, the real-time requirements of the current task, and the computing resources of the AI ​​gateway in real time during the operation of the AI ​​gateway. Based on the monitoring results, it makes adaptive reasoning strategy decisions and then implements real-time resource scheduling based on task priority and computing resources. The inference optimization module is used to select a lightweight inference engine that is compatible with the hardware platform of the AI ​​gateway, and to compile and optimize the model computation graph. It merges continuous operators, combining the continuous operations of convolution, batch normalization and activation functions into a single operator to reduce the number of memory accesses. Then, it uses heterogeneous computing technology to reasonably allocate computing tasks to different hardware units of CPU, GPU and NPU according to task type, and realizes efficient task splitting and collaborative processing through OpenCL / OpenVX interface. The data processing module is used to preprocess the data received by the AI ​​gateway by denoising and feature dimensionality reduction, and to establish a local data caching mechanism to store high-frequency input sample features. At the same time, it adopts an incremental learning algorithm so that when the local data distribution changes, only the affected model parameters are updated. The collaborative reasoning module is used to build a hierarchical reasoning architecture and adopts a combination of federated learning and model distillation to achieve task allocation and data interaction between the edge and the cloud.

8. The real-time dynamic inference optimization system for AI gateways according to claim 7, characterized in that, Lightweight network architectures include the MobileNet series, ShuffleNet, and YOLO-Lite; Pruning refers to the pruning operation performed through L1 / L2 regularization to remove redundant connections and reduce parameters in the model. Quantization refers to converting model parameters from high-bit to low-bit, thereby reducing memory usage and computational load. Knowledge distillation refers to using teacher models to guide student model learning, thereby reducing the model size while ensuring model accuracy.

9. The real-time dynamic inference optimization system for AI gateways according to claim 7, characterized in that, The utilization of computing resources includes the utilization rate of CPU, GPU, and NPU, memory usage, and queue length; The adaptive inference strategy decision-making content is as follows: if the simplicity of the input data is lower than the set threshold and the real-time requirement is higher than the set threshold, then a lightweight model is used for inference first; if the simplicity of the input data is higher than the set threshold, then a high-precision model is used when computing power is sufficient, and the model is automatically downgraded when computing power is insufficient; for models with multi-branch structures, an early stopping mechanism is adopted to output results in advance, and results are output in advance when simple samples meet specific conditions to reduce unnecessary calculations. Real-time resource scheduling based on task priority and computing resources involves prioritizing different tasks. In industrial scenarios, fault warning tasks are given higher priority than regular inspection tasks, and computing resources are allocated to them first. Model operating parameters are dynamically adjusted based on computing power usage. These parameters include inference frequency and data processing batch size.

10. The real-time dynamic inference optimization system for AI gateways according to any one of claims 7 to 9, characterized in that, The hardware-adaptive lightweight inference engine includes deploying TensorRT on the NVIDIA GPU platform, leveraging TensorRT's support for FP16 / INT8 quantization and graph optimization features to accelerate model inference; or, Leverage the cross-platform and lightweight advantages of using ONNX Runtime or TFLite on ARM architecture devices; Data preprocessing employs a spatiotemporal filtering algorithm to denoise the data collected by industrial sensor equipment, and utilizes PCA and LDA methods to perform feature dimensionality reduction on high-dimensional data, thereby reducing invalid data and computational load. The local data caching mechanism works as follows: when new data is input, it is first compared with cached features to quickly determine whether a conclusion can be drawn directly, thus reducing the number of full inferences performed by the model.

Images