An edge NPU-based online model updating method and system

Abstract

The application relates to the technical field of edge computing, and discloses an online model updating method and system based on an edge NPU, aiming to solve the problem that a model deployed on an edge device is difficult to be efficiently and adaptively updated under a dynamic working condition when resources are limited. The method performs normal inference on the edge NPU, and evaluates the deviation of a model prediction result from an actual working condition through a performance monitoring module; when performance decline is detected and an updating threshold is triggered, an online updating process is started. The process adopts a resource-aware NPU computing power scheduling strategy, dynamically allocates NPU computing cores without interrupting normal inference, and after a precompiled lightweight updating graph is spliced to an inference graph in the memory by using an AI compiler, fast fine-tuning is realized without reloading the model, and edge side online updating is completed. The application also discloses a system for implementing the above method, which can improve the adaptive capability of an AI model and guarantee the real-time performance of an inference task.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to the deployment and optimization of edge computing and deep learning models, and particularly to a method and system for online model updates on an edge NPU (neural network processing unit). Background Technology

[0002] In recent years, with the development of artificial intelligence technology, deep learning models have been increasingly widely used in fields such as industrial manufacturing, for example, for parameter optimization in die-casting processes. To meet real-time requirements, these complex AI models are increasingly migrating from the cloud to edge computing devices, such as edge computing platforms equipped with NPUs (like NVIDIA Jetson series or Huawei Atlas series).

[0003] However, AI models deployed at the edge face significant challenges: industrial conditions are constantly changing. For example, in die-casting processes, mold changes, equipment wear, or alterations in environmental parameters can all lead to a decrease in the accuracy of the original model's predictions. The traditional solution is to transmit all data collected under new conditions back to the cloud, where engineers retrain a "Learn Specialist" model. This process is time-consuming and labor-intensive, failing to meet the high-efficiency production goals that modern industry desires, such as "one-time trial molds."

[0004] Another approach is to give edge models a degree of adaptability, known as "Fine-tune Generalist" models. These models can be rapidly fine-tuned locally using a small amount of new data collected in the target environment to adapt to new conditions. However, performing model training or fine-tuning on resource-constrained devices like edge NPUs faces strict limitations in computing power, memory, and power consumption.

[0005] Therefore, the industry urgently needs an online model update method that can be executed locally on the edge NPU, with low resource consumption and fast response speed, in order to solve the problem of edge AI models being "unsuited" under dynamic working conditions. Summary of the Invention

[0006] This invention aims to solve the technical problem that existing edge AI models struggle to achieve fast and efficient adaptive updates on resource-constrained NPUs when facing dynamically changing operating conditions.

[0007] To achieve the above objectives, this invention provides an online model update method based on an edge NPU, characterized by comprising the following steps: Step S1: Real-time Inference. Execute an initial deep learning model on the NPU of the edge computing device, and perform normalized inference based on real-time operational data to obtain prediction results.

[0008] Step S2: Performance Monitoring. A performance monitoring module is set up to compare the predicted results with actual operating condition feedback data (e.g., the trial molding effect in the die-casting process) in real time to evaluate the current performance of the initial model.

[0009] Step S3: Trigger Update. When the performance monitoring module detects a decline in model performance and triggers a preset update threshold (e.g., the number of trial runs required to adapt to new operating conditions exceeds a preset value), a resource-aware online update process is initiated locally on the edge computing device.

[0010] Step S4: Resource-Aware Online Fine-Tuning. The online update process first monitors the NPU computing load of the normalized real-time inference task in S1, and dynamically allocates the AI ​​computing cores of the edge NPU according to the preset scheduling strategy without interrupting the real-time inference task. This update calculation uses an AI compiler to dynamically concatenate a pre-compiled lightweight "update graph" (including backpropagation and optimizer steps) to the "inference graph" already residing in the NPU memory, realizing fine-tuning calculation without reloading the model, thereby generating and deploying an updated model on the edge.

[0011] This invention also provides an online model update system based on an edge NPU, characterized by comprising: an edge AI chip configured with an NPU; a model inference module running on the NPU for performing routine real-time inference based on real-time operating data; a performance monitoring module for comparing the prediction results with actual operating feedback data and determining whether the model performance has degraded; and an online update scheduling module activated when receiving an update signal triggered by the performance monitoring module. This module first monitors the NPU computing load of the model inference module and, according to a preset scheduling strategy, dynamically allocates the AI ​​computing cores of the NPU without interrupting the real-time inference to execute a parameter-efficient fine-tuning strategy, generating and redeploying an updated model.

[0012] The beneficial effects of this invention are as follows: Improved model adaptability: By performing online fine-tuning locally at the edge, the model can quickly adapt to changing operating conditions and equipment wear, improving robustness. Reduced cloud dependency and data transmission costs: Model updates are completed locally at the edge, eliminating the need to transmit large amounts of (potentially privacy-sensitive) production data back to the cloud, reducing latency and bandwidth costs. Guaranteed real-time performance of core business operations: Through resource-aware computing power scheduling, update tasks are executed without interrupting core inference tasks, avoiding production delays caused by edge computing resource conflicts. Extremely low update latency: Through dynamic graph stitching technology, the significant time and memory overhead required for unloading the "inference graph" and loading the "training graph" is avoided, achieving truly "online" rapid updates. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the online model update method based on an edge NPU as described in this invention. The diagram shows the process from real-time operational data input, through model inference and performance monitoring, to triggering the online update scheduling module. It explains how this module performs NPU computing power awareness and dynamic graph stitching, and executes fine-tuning without interrupting inference, ultimately completing a closed-loop process. Figure 2 This is a system architecture block diagram of an online model update system based on an edge NPU as described in this invention. The diagram illustrates a full-stack AI system architecture, including a bottom-level edge AI chip layer (NPU), an intermediate compilation layer (containing an AI compiler) and framework layer, and a top-level application layer. The online update scheduling module of this invention works collaboratively within this architecture. Figure 3 This is a conceptual diagram illustrating the "Fine-tune Generalist" strategy employed in this invention. The diagram demonstrates how, when the amount of data from the target environment is limited, the fine-tuning strategy employed in this invention outperforms the "Learn Specialist" strategy, which requires a large amount of data for training, thereby achieving efficient adaptation. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0016] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

[0017] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to an embodiment of an online model update method and system based on an edge NPU in this application, which includes: Example 1: An Online Model Update Method Based on Edge NPU Reference Figure 1 This embodiment discloses a method for implementing adaptive model updates on edge computing devices. This method aims to address the problem of performance degradation of fixed models caused by dynamic changes in operating conditions (such as mold replacement or equipment wear) in industrial settings (e.g., die-casting processes).

[0018] Step S1: Real-time Inference In one specific embodiment, an initial deep learning model is deployed on the NPU (Neural Processing Unit) of an edge computing device. This edge computing device can be an edge computing platform equipped with an AI chip (such as the NVIDIA Jetson series or Huawei Atlas 200 AI acceleration module). The model inference module utilizes the NPU's AI computing core to process real-time operating data from the industrial site and output prediction results, such as process parameters for die-casting control.

[0019] Step S2: Performance Monitoring The system includes a performance monitoring module that runs continuously to evaluate the model's actual performance under current operating conditions. This module compares the "predicted results" output from S1 with "actual operating condition feedback data" from the production line in real time. In the die-casting process, a key feedback data point is the "trial molding effect." For example, the monitoring module will calculate how many trial moldings the model needs under new operating conditions to achieve acceptable product quality.

[0020] Step S3: Trigger Update The system presets an update threshold. This threshold can be set according to specific industrial goals. For example, in a high-efficiency production scenario that pursues "one-time trial mold," the threshold can be set to "number of trial molds > 1." When the S2 performance monitoring module detects a decline in model performance and triggers this threshold (for example, adapting to a new mold requires 3-5 trial molds), the system determines that the current model can no longer adapt to the new working conditions and immediately initiates a resource-aware online update process locally on the edge computing device.

[0021] Step S4: Online fine-tuning of resource awareness This step aims to solve two major technical challenges when performing "inference" and "lightweight training" simultaneously on an edge NPU: (1) NPU resource conflict: the fine-tuning task (S4) will preempt the computing power of the normal inference task (S1), causing critical production (such as die casting control) tasks to be stuck; (2) Model reload delay: switching from the "inference graph" of S1 to the "training graph" of S4 requires unloading and reloading the model, which takes a lot of time and consumes a lot of memory on edge devices.

[0022] The technical solution of this invention solves the above-mentioned problems through the following combination of NPU computing power scheduling and dynamic graph stitching: NPU Computing Power Scheduling: When S3 is triggered, the "Online Update Scheduling Module" is activated. This module first monitors the NPU computing power load of the S1 normalized inference task, and dynamically allocates the AI ​​computing cores of the NPU according to the preset scheduling strategy without interrupting the S1 task.

[0023] Dynamic Graph Stitching: This invention utilizes an AI compiler to address the overload latency issue. The AI ​​compiler does not compile a complete training graph (including forward, backward, and optimization). Instead, leveraging the fact that the "inference graph" (i.e., the forward propagation graph) of S1 already resides in the NPU's memory, the AI ​​compiler only compiles the backpropagation computation and optimizer steps (such as Adam updates) required for efficient parameter fine-tuning strategies (e.g., the "Fine-tune Generalist" strategy, or more specifically LoRA, Adapter) into a lightweight, pre-compiled update graph. When S4 performs fine-tuning, the online update scheduling module dynamically stitches this "update graph" to the end of the S1 "inference graph" in memory, forming a temporary, complete training graph. After executing the forward propagation (inference) of S1, the NPU seamlessly executes the computation of the "update graph" (backpropagation and parameter updates). This computation utilizes a small amount of target environment data collected under the new operating conditions. After fine-tuning, the "update graph" is dynamically separated.

[0024] In one specific embodiment, the process of performing step S1 may specifically include the following steps: (1) Deploy an initial deep learning model on an edge computing device, preferably an AI edge computing platform equipped with an NPU.

[0025] (2) The inference module of the model is configured to call the AI ​​computing core of the NPU.

[0026] (3) Receive real-time operating data from the industrial site (e.g., sensor data from the die-casting machine), perform forward inference, and output prediction results (e.g., process parameters for die-casting control).

[0027] In one specific embodiment, the process of performing step S2 may specifically include the following steps: (1) Start a performance monitoring module that continuously compares the “predicted results” output in S1 with the “actual operating condition feedback data”.

[0028] (2) In the application scenario of die casting process, the "actual working condition feedback data" specifically refers to "trial mold effect". For example, the monitoring module will count how many trial molds (such as 3-5 times) are needed for the model to reach the qualified standard under new working conditions or new mold.

[0029] In one specific embodiment, the process of performing step S3 may specifically include the following steps: (1) Preset an update threshold based on the production target (e.g., to achieve “one trial run”).

[0030] (2) When the performance monitoring module of S2 detects a decline in model performance and triggers this threshold (for example, the actual number of model trials > 1), the system determines that the current model can no longer adapt to the new working conditions.

[0031] (3) The system immediately initiates the online update process locally on the edge computing device without having to send a large amount of data back to the cloud.

[0032] In one specific embodiment, the process of performing step S4 may specifically include the following steps: (1) Reference Figure 3 The online update process employs a "Fine-tune Generalist" strategy. This strategy uses only a small amount of target environment data collected by S2 under new operating conditions.

[0033] (2) The fine-tuning strategy is designed to be parameter-efficient, for example, updating only a small portion of the parameters of the initial model.

[0034] (3) The online update process is executed by an "online update scheduling module". Before performing the calculation, the module first monitors the NPU computing load of the S1 normalized inference task and dynamically allocates NPU resources according to the preset scheduling strategy without interrupting the S1 task.

[0035] (4) In a preferred embodiment, the preset scheduling strategy is selected from: a) Time-sharing multiplexing strategy: The online update scheduling module monitors the computing idle window of the S1 task and performs the fine-tuning update calculation in (2) within the window period; b) Computing power partitioning strategy: The online update scheduling module dynamically divides the AI ​​computing core of the NPU into an "inference computing area" and a "training computing area". The inference computing area ensures the real-time performance of the S1 task, and the training computing area performs fine-tuning update calculation in the background.

[0036] (5) Reference Figure 2 The AI ​​system full-stack architecture shown calls an AI compiler, which is used to compile the parameter efficient fine-tuning strategy described in (2) into a lightweight "update graph" containing only backpropagation and parameter optimization steps during compilation.

[0037] (6) During runtime, the “online update scheduling module” performs dynamic graph splicing on the NPU resources allocated in (4), and dynamically attaches the “update graph” generated in (5) to the S1 “inference graph” (i.e. forward propagation graph) that has already resided in the NPU memory, forming a temporary and complete training graph.

[0038] (7) The computation of the temporary training graph generated in (6) is accelerated by utilizing the AI ​​computing core of the edge NPU.

[0039] (8) Reference Figure 1 After the calculation is completed, the "updated graph" is dynamically separated. Since the model parameters are updated in-place in memory, the system does not need to reload or replace the model. In the next inference cycle, the model inference module of S1 will automatically use the updated parameters, thus forming a closed loop.

[0040] Example 2: An Online Model Update System Based on Edge NPU Reference Figure 2 This embodiment also discloses a system for implementing the above method. This system is an edge computing system integrating a full-stack AI architecture, characterized by comprising: An edge AI chip: serving as the hardware foundation of the system, providing core AI computing power. In a preferred embodiment, this chip is an NVIDIA Jetson Xavier NX module or a Huawei Atlas 200 AI acceleration module, which integrates an NPU.

[0041] A model inference module: running on top of framework layers (such as PyTorch, TensorFlow) and AI inference engines (such as TensorRT), calling the NPU to execute S1's real-time inference tasks.

[0042] A performance monitoring module: an application-layer software module used to perform S2 comparison and evaluation tasks.

[0043] An online update scheduling module: a core component of this system. This module is activated and configured when it receives an update signal triggered by the performance monitoring module. Perform computing power scheduling: First, monitor the NPU computing power load of the model inference module, and dynamically allocate NPU resources according to the preset scheduling strategy (such as time-sharing multiplexing or computing power partitioning); Perform dynamic graph stitching: Call the AI ​​compiler to perform dynamic graph stitching, which appends a pre-compiled "update graph" to the "inference graph" in memory, enabling fine-tuning computation without reloading; Redeployment: After the parameters are updated, this module enables the model inference module to immediately use the new parameters without reloading.

Claims

1. An edge NPU-based online model updating method and system, characterized in that, Includes the following steps: S1: Execute an initial deep learning model on the NPU of the edge computing device to perform normalized real-time inference to obtain prediction results; S2: Set up a performance monitoring module to compare the prediction results with the actual working condition feedback data (such as the die-casting mold test results) in real time to evaluate the current performance of the initial model; S3: When the performance monitoring module detects a decline in model performance and triggers a preset update threshold (such as when the number of model trials exceeds a preset value), a resource-aware online update process is initiated locally on the edge computing device. S4: The online update process first monitors the NPU computing load of the normalized real-time inference task in S1, and dynamically allocates the AI ​​computing core resources of the edge NPU according to the preset scheduling strategy without interrupting the real-time inference task, so as to perform a parameter-efficient fine-tuning update calculation, thereby generating and deploying an updated model on the edge side.

2. The method according to claim 1, characterized in that: The preset update threshold in S3 is set according to the goal of achieving "one trial run" in the die casting process. When the number of trial runs required to adapt to the new parameters exceeds one, an update is triggered.

3. The method according to claim 1, characterized in that: The parameter-efficient fine-tuning strategy in S4 is a "Fine-tune Generalist" strategy, which uses a small amount of newly acquired target environment data on the edge computing device to fine-tune the model.

4. The method according to claim 1, characterized in that: The update calculation in S4 utilizes an AI compiler to compile the parameter fine-tuning strategy into a lightweight "update graph" containing only backpropagation and parameter optimization steps during the compilation phase. When S4 is executed, the online update process dynamically splices this "update graph" after the "inference graph" already residing in the NPU memory for S1, forming a temporary, complete training graph, thereby completing the fine-tuning calculation without reloading the model.

5. The method according to claim 1, characterized in that: The preset scheduling strategy in S4 is selected from at least one of the following: a) Time-sharing multiplexing strategy: The online update process monitors the computing idle window of the real-time inference task of S1 and performs fine-tuning update calculations within the window period; b) Computing power partitioning strategy: The online update process dynamically divides the AI ​​computing core of the NPU into an inference computing area and a training computing area. The inference computing area ensures the real-time performance of the S1 task, and the training computing area performs fine-tuning update calculations in the background.

6. The method according to claim 1, characterized in that: The edge NPU is either an NVIDIA Jetson series or a Huawei Atlas series edge AI chip.

7. An online model update system based on an edge NPU, characterized in that, include: An edge AI chip is configured with an NPU; a model inference module runs on the NPU and is used to perform normalized real-time inference based on real-time operating data. A performance monitoring module is used to compare the prediction results with actual working condition feedback data and determine whether the model performance has degraded; an online update scheduling module is activated when it receives an update signal triggered by the performance monitoring module. This module first monitors the NPU computing load of the model inference module and, according to a preset scheduling strategy, dynamically allocates the AI ​​computing cores of the NPU without interrupting the real-time inference, so as to execute a parameter-efficient fine-tuning strategy, generate and redeploy an updated model.

8. The system according to claim 7, characterized in that: The online update scheduling module further includes an AI compiler, which is used to compile the computational task of the fine-tuning strategy into a lightweight "update graph" that only contains backpropagation and parameter optimization steps. This allows the online update scheduling module to dynamically append the "update graph" to the "inference graph" already occupied by the model inference module when performing an update, thereby achieving update computation without reloading the model.

9. The system according to claim 7, characterized in that: The edge AI chip is either an NVIDIA Jetson series module or a Huawei Atlas 200 AI acceleration module.

Images