Complex scene-oriented AI large model lightweight deployment method

Abstract

The invention provides a complex scene-oriented AI large model lightweight deployment method, and relates to the technical field of edge computing, and the method comprises the steps: carrying out the structured pruning of a pre-trained Transform network based on the attention head importance score, carrying out the dynamic sparsification of the activation state of a feedforward network according to the input tensor entropy value, employing the dynamic mixing precision quantization, and carrying out the reconstruction of an AI large model. Obtaining network parameters after pruning quantization; deploying the pruned and quantized network parameters to an edge computing device, distributing a feature extraction operator to a neural network processor through a heterogeneous computing scheduler, and unloading a classification operator to a multi-core central processing unit; and managing an on-chip memory in combination with a virtual memory paging mechanism, realizing zero-copy data transmission by utilizing a direct memory access controller, and outputting a reasoning result tensor. According to the method, efficient and reliable operation of the large model at the resource-constrained edge node is realized.

Description

Technical Field

[0001] This invention relates to the field of edge computing technology, and in particular to a lightweight deployment method for large AI models in complex scenarios. Background Technology

[0002] With the deep integration of the Industrial Internet of Things (IIoT) and intelligent manufacturing, large-scale smart factories have entered a new stage of multi-device collaboration and end-to-end digitalization. The stable operation of thousands of heterogeneous devices within the factory directly determines production efficiency, and real-time diagnosis and early warning of equipment faults have become a core link in ensuring the continuous operation of the production line. Currently, fault diagnosis mainly relies on the analysis of multi-source heterogeneous data, including mechanical vibration signals collected by vibration sensors, component appearance images captured by high-definition industrial cameras, and equipment operating temperatures recorded by temperature sensors. AI models are needed to achieve multi-modal feature fusion and accurate identification.

[0003] In existing technologies, lightweight deployment methods for large models mainly focus on model compression and edge computing adaptation, but they still have significant limitations in complex industrial scenarios: the scenario adaptability of model compression strategies is insufficient; traditional structured pruning is mostly based on feature importance scoring of general datasets, which does not fully consider "edge cases" of industrial equipment failures (such as low-probability but high-risk failure modes, such as abnormal vibrations of a robotic arm at a specific angle), which can easily lead to the network structure corresponding to such key features being mistakenly pruned, resulting in an increase in the failure detection rate under extreme working conditions. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a lightweight deployment method for large AI models in complex scenarios, so as to achieve efficient and reliable operation of large models on resource-constrained edge nodes.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a lightweight deployment method for large AI models in complex scenarios, the method comprising: Step 1: Perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head, dynamically sparsify the activation state of the feedforward network according to the input tensor entropy value, and use dynamic mixed precision quantization to obtain the pruned and quantized network parameters. Step 2: Deploy the pruned and quantized network parameters to the edge computing device. Use the heterogeneous computing scheduler to allocate the feature extraction operator to the neural network processor and the classification operator to the multi-core central processing unit. Combine the virtual memory paging mechanism to manage the on-chip memory and use the direct memory access controller to achieve zero-copy data transmission and output the inference result tensor. Step 3: When the inference result tensor is detected to contain a preset abnormal feature vector, differential privacy gradient synchronous update is performed by minimizing the difference between the probability distribution of the edge device output and the teacher model output to obtain the updated network parameters. Step 4: Based on the updated network parameters, real-time calculation load matrix, task-dependent data and time delay constraints, map the edge node computing power, memory and network input / output resources into a multi-dimensional constraint space; recursively construct a multi-dimensional tree index to locate partitions that meet the time delay, and use an approximate nearest neighbor search to match the Pareto final resource allocation. Step 5: Based on the final resource allocation, dynamically allocate computing instance resources to maintain end-to-end latency within the service level agreement threshold.

[0006] Furthermore, structured pruning is performed on the pre-trained Transformer network based on the importance score of the attention head. The activation state of the feedforward network is dynamically sparsified according to the input tensor entropy value, and dynamic mixed-precision quantization is used to obtain the pruned and quantized network parameters, including: Step 1.1: Perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head to generate the pruned network model; Step 1.2: Input the pruned network model into the feedforward network, and dynamically sparsify the activation state according to the entropy value of the input tensor to generate a feedforward sub-network with sparse activation mode. Step 1.3: Perform dynamic mixed-precision quantization on the weight tensors of the feedforward subnetwork with sparse activation mode, adaptively allocate bit width based on layer sensitivity analysis, and generate lightweight network parameters with three-stage compression.

[0007] Furthermore, the pruned and quantized network parameters are deployed to edge computing devices. A heterogeneous computing scheduler allocates feature extraction operators to neural network processors, while classification operators are offloaded to multi-core CPUs. On-chip memory is managed using a virtual memory paging mechanism, and zero-copy data transfer is achieved using a direct memory access controller. The output inference result tensor includes: Step 2.1: In response to the deployment command, load the pruned and quantized network parameters into the heterogeneous memory architecture of the edge computing device; Step 2.2: Based on the loaded network parameters, the topology of the model computation graph is parsed by the heterogeneous computing scheduler. The feature extraction operator is assigned to the neural network processor according to the parsing result, and the classification operator is offloaded to the multi-core central processing unit according to the parsing result. Step 2.3: For the feature extraction operators allocated to the neural network processor, the access frequency of the model parameters associated with the operators is analyzed according to the virtual memory paging mechanism, and the high-frequency parameter pages are dynamically cached to the on-chip static memory based on the access frequency analysis results. Step 2.4: After the neural network processor completes the execution of the feature extraction operator, it transmits the generated feature tensor directly to the memory space of the central processing unit through the direct memory access controller, according to the state of the cached data pages, via physical address mapping. Step 2.5: After receiving the transmitted feature tensor, the central processing unit executes the unloaded classification operator to generate the inference result tensor, and outputs the inference result tensor through the edge device.

[0008] Furthermore, when the inference result tensor is detected to contain a preset abnormal feature vector, differential privacy gradient synchronous updates are performed by minimizing the difference in probability distribution between the edge device output and the teacher model output, resulting in updated network parameters, including: Step 3.1: After receiving the inference result tensor output by the edge device, detect whether the tensor contains a preset abnormal feature vector; Step 3.2: When an abnormal feature vector is detected, the teacher model is invoked to perform inference based on the input data that triggered the anomaly detection, and the corresponding baseline probability distribution is generated. Step 3.3: Based on the baseline probability distribution, calculate the KL divergence between the edge device output probability distribution and the baseline probability distribution; Step 3.4: Using the KL divergence value as the optimization objective, generate a gradient update instruction and add Laplace noise to this instruction; Step 3.5: Synchronize the noise gradient update command to the edge device and control the edge device to apply the command to update the network parameters.

[0009] Furthermore, based on the updated network parameters, real-time computation load matrix, task-dependent data, and time-delay constraints, the computing power, memory, and network input / output resources of edge nodes are mapped into a multi-dimensional constraint space, including: Step 4.1: Receive the updated network parameters, synchronously parse the computational load matrix and task dependency data reported in real time by the edge nodes, generate the execution order topology data between operators, and extract the maximum latency threshold of the critical path by combining latency constraints. Step 4.2 transforms the critical path latency threshold into resource dimension constraints and constructs a three-dimensional resource space coordinate system. The computing power axis is normalized according to the proportion of actual available floating-point computing power of the neural network processor and the central processing unit; the memory axis maps the on-chip static memory and dynamic random access memory capacity into a continuous virtual address space based on a hierarchical storage architecture; the network input / output axis uses the product of transmission bandwidth and the reciprocal of end-to-end latency as the quantization benchmark. Step 4.3: Based on the updated network parameters and the computational load matrix, calculate the resource demand vector and project it onto the established three-dimensional space. The computational power demand vector is determined by the peak computational cost of the model operators; the memory demand vector is determined by the sum of the model parameter volume and the peak occupancy of the activation tensor; the network input and output demand vector is determined by dividing the amount of data transmitted by the feature tensor by the critical path delay threshold. Construct a multi-dimensional hypercube with delay constraints in the three-dimensional space to complete the mapping of resources to the constraint space.

[0010] Furthermore, a multidimensional tree index is recursively constructed to locate partitions that meet the latency requirements, and an approximate nearest neighbor search is used to match the Pareto final resource allocation, including: Step 4.4: Using the multidimensional constraint space as input, recursively perform spatial bisection partitioning along the direction of the normal vector of the time delay constraint hyperplane, calculate the variance of each resource dimension, and select the dimension with the largest variance as the splitting axis; use the median of the resource vectors in the current partition as the cutting point. Step 4.5: Perform latency compliance verification on the leaf partitions generated by recursive partitioning, mark the partitions that meet the latency threshold as feasible subspaces, construct a locality-sensitive hash index table within the feasible subspaces, and use the linear combination of resource utilization and the reciprocal of latency as the hash key; bucket the vectors according to the Pareto front distance. Step 4.6: Input the real-time resource demand vector into the hash index table, perform an approximate nearest neighbor search, minimize the weighted sum of resource idle rate and latency margin; extract the non-dominated solution set from the same hash bucket, sort the candidate solution set by Pareto order, and select the highest-order solution as the final resource allocation scheme.

[0011] Furthermore, based on the final resource allocation, computing instance resources are dynamically allocated to maintain end-to-end latency within the service level agreement threshold, including: Step 5.1: Receive the Pareto final resource allocation scheme and parse the computing power allocation ratio, memory space division and network input / output bandwidth quota in the scheme to generate parsed resource allocation data; Step 5.2: Based on the resource allocation data, calculate the runtime constraints of the instance, allocate the neural network processor computing power quota and the number of on-chip static memory page frames to the feature extraction operator instance; allocate the central processing unit core binding strategy and dynamic random access memory limit to the classification operator instance; and set the inter-container communication bandwidth threshold based on the quantization value of the network input and output axes. Step 5.3: Based on the set runtime constraints, dynamically monitor the end-to-end inference latency of edge devices; when the real-time latency deviates from the service level agreement threshold, trigger the elastic scaling operation of the container instance; Step 5.4: Based on the elastic scaling operation, the updated network parameters are injected in real time through the container runtime interface to ensure the consistency of the model state during elastic scaling, thereby keeping the end-to-end inference latency within the service level agreement threshold range.

[0012] Secondly, lightweight deployment systems for large AI models in complex scenarios include: The acquisition module is used to perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head, dynamically sparsify the activation state of the feedforward network according to the input tensor entropy value, and use dynamic mixed precision quantization to obtain the pruned and quantized network parameters. The inference module is used to deploy the pruned and quantized network parameters to edge computing devices, allocate feature extraction operators to neural network processors through a heterogeneous computing scheduler, and offload classification operators to multi-core central processing units; it manages on-chip memory using a virtual memory paging mechanism, achieves zero-copy data transfer using a direct memory access controller, and outputs inference result tensors. The computation module is used to perform differential privacy gradient synchronous updates when the inference result tensor is detected to contain a preset abnormal feature vector. This is done by minimizing the difference in probability distribution between the edge device output and the teacher model output to transfer knowledge and obtain updated network parameters. Based on the updated network parameters, the real-time computation load matrix, task-dependent data, and time delay constraints, the computing power, memory, and network input / output resources of the edge nodes are mapped into a multi-dimensional constraint space. A multi-dimensional tree index is recursively constructed to locate partitions that meet the time delay requirements, and an approximate nearest neighbor search is used to match the Pareto final resource allocation. The processing module is used to dynamically allocate computing instance resources based on the final resource allocation ratio, and maintain end-to-end latency within the service level agreement threshold.

[0013] Thirdly, a computing device includes: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0014] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0015] The above-described solution of the present invention has at least the following beneficial effects: Through multi-stage collaborative design, this solution provides an efficient solution for lightweight deployment of large AI models in complex scenarios: it combines attention head importance pruning, dynamic sparsification of input tensor entropy values, and dynamic mixed-precision quantization to achieve significant model lightweighting while retaining key features and reducing the risk of edge case loss; through heterogeneous computing scheduling, virtual memory paging management, and zero-copy transmission, it improves the utilization of edge device computing power and reduces inference latency to meet real-time requirements; with the help of knowledge transfer based on KL divergence and differential privacy gradient updates, it enables dynamic adaptation of edge models to abnormal scenarios while ensuring data security; and through the construction of multi-dimensional resource constraint space, Pareto final allocation search, and elastic scaling of containers, it dynamically responds to load fluctuations and ensures that end-to-end latency remains stable within the service level agreement threshold. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a lightweight deployment method for large AI models in complex scenarios, provided by an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of a lightweight deployment system for large AI models in complex scenarios provided by an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0019] like Figure 1 As shown, embodiments of the present invention propose a lightweight deployment method for large AI models in complex scenarios, the method comprising the following steps: Step 1: Perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head, dynamically sparsify the activation state of the feedforward network according to the input tensor entropy value, and use dynamic mixed precision quantization to obtain the pruned and quantized network parameters. Step 2: Deploy the pruned and quantized network parameters to the edge computing device. Use the heterogeneous computing scheduler to allocate the feature extraction operator to the neural network processor and the classification operator to the multi-core central processing unit. Combine the virtual memory paging mechanism to manage the on-chip memory and use the direct memory access controller to achieve zero-copy data transmission and output the inference result tensor. Step 3: When the inference result tensor is detected to contain a preset abnormal feature vector, differential privacy gradient synchronous update is performed by minimizing the difference between the probability distribution of the edge device output and the teacher model output to obtain the updated network parameters. Step 4: Based on the updated network parameters, real-time calculation load matrix, task-dependent data and time delay constraints, map the edge node computing power, memory and network input / output resources into a multi-dimensional constraint space; recursively construct a multi-dimensional tree index to locate partitions that meet the time delay, and use an approximate nearest neighbor search to match the Pareto final resource allocation. Step 5: Based on the final resource allocation, dynamically allocate computing instance resources to maintain end-to-end latency within the service level agreement threshold.

[0020] In this embodiment of the invention, through the collaborative compression of structured pruning, dynamic sparsity, and mixed precision quantization, the model parameter scale is significantly reduced while retaining key features, adapting to the resource-constrained characteristics of edge devices. Heterogeneous computing scheduling, virtual memory management, and zero-copy transmission are used to improve the utilization rate of edge device computing power and reduce data transmission time, ensuring real-time inference. Knowledge transfer based on probability distribution differences and differential privacy gradient updates enable dynamic adaptation of edge models to abnormal scenarios while protecting data security. Combining multi-dimensional resource constraint space construction and Pareto final allocation search, the computing power, memory, and network resources of edge nodes are accurately matched to dynamically respond to load fluctuations. Finally, through container orchestration and elastic scaling, end-to-end latency is ensured to remain stable within the service level agreement threshold, improving the efficiency of deploying large AI models at the edge in complex scenarios.

[0021] In a preferred embodiment of the present invention, step 1 above may include: Step 1.1: Perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head to generate the pruned network model; Step 1.2: Input the pruned network model into the feedforward network, and dynamically sparsify the activation state according to the entropy value of the input tensor to generate a feedforward sub-network with sparse activation mode. Step 1.3: Perform dynamic mixed-precision quantization on the weight tensors of the feedforward subnetwork with sparse activation mode, adaptively allocate bit width based on layer sensitivity analysis, and generate lightweight network parameters with three-stage compression.

[0022] In this embodiment of the invention, a three-stage collaborative compression strategy achieves precise lightweighting of large AI models: structured pruning based on attention head importance scoring selectively preserves network structures that play a decisive role in key features, avoiding redundant and ineffective parameters; combined with dynamic sparse activation states of input tensor entropy values, the computational intensity can be adaptively adjusted according to feature importance, reducing redundant computation while retaining key information with high entropy values; and dynamic mixed-precision quantization based on layer sensitivity analysis, by allocating high bit widths to high-sensitivity layers and low bit widths to low-sensitivity layers, maximizes the compression of weight tensor size while maintaining inference accuracy. This three-stage compression mechanism is formed through the synergy of these three elements.

[0023] In this embodiment of the invention, the specific steps include: Step 1.1, Structured Pruning Stage: By quantifying the importance of each attention head, remove network components (such as weight matrices and connection layers) corresponding to low-importance heads, simplifying the model structure while retaining the core attention mechanism.

[0024] Activation sparsification stage: The pruned model is connected to the feedforward network, and the activation mode of the neurons in the feedforward network is dynamically adjusted according to the feature complexity (entropy value) of the input data. Only the activation calculation of high information density regions is retained to form a sparsified sub-network.

[0025] Hybrid precision quantization stage: For the weight parameters of the sparse subnetwork, different quantization bit widths are allocated according to the degree of influence (sensitivity) of each layer on the inference accuracy. High-precision parameters are deployed in a mixed manner with low-precision parameters to finally form a lightweight model.

[0026] Calculate the importance score of each attention head: For each attention head, calculate the Pearson correlation coefficient between its output features and the final inference result of the model, as well as its contribution to the decrease of the loss function during training. Then, sum the two values ​​by weighting them according to the preset weight coefficients 1 and 2 to obtain the importance score of the attention head. Pruning operation: Set an importance score threshold, filter out attention heads with scores below the threshold, remove the weight matrices (query, key, value matrices and output projection matrices) and connection layers corresponding to these attention heads, and retain the attention heads with scores above the threshold and their network connections to generate the pruned network model.

[0027] Step 1.2, Calculate the input tensor entropy: For the input tensor of the feedforward network (from the output features of the pruned model), count the frequency (or normalized relative value) of each element in the tensor as the element probability value. For each probability value, calculate "probability value × logarithm of the probability value to base 2", then sum all the results after taking the negative value to obtain the entropy value of the input tensor; Determine the sparsification threshold: Calculate the threshold according to the formula "sparserization threshold = (average entropy value of input tensor × scaling factor) + minimum retention threshold" (the scaling factor and minimum retention threshold are set according to the scenario requirements); Dynamically sparse activation state: Divide the input tensor into local regions and calculate the local entropy value of each region. If the local entropy value is lower than the sparsification threshold, then set the activation value of the feedforward network neuron corresponding to that region to zero (or stop the activation calculation); if the local entropy value is higher than the threshold, then retain the activation state to form a feedforward sub-network with a sparse activation mode.

[0028] Step 1.3, Calculate the layer sensitivity score: For each layer of the feedforward sub-network, perform inference using 8-bit quantization and 32-bit floating-point calculation respectively, and obtain the precision difference between the two (precision loss value). Calculate the sensitivity score of the layer according to the formula "Sensitivity score = (precision loss value ÷ 32-bit floating-point inference precision) × 100%"; Allocate quantization bit width: Determine the maximum sensitivity score among all layers, and allocate the bit width to each layer according to the formula "Quantization bit width = base bit width (4 bits) + (sensitivity score of this layer ÷ maximum sensitivity score) × bit width adjustment range (e.g., 0-12 bits)" (ensuring that the bit width of high-sensitivity layers is not lower than the preset minimum value); Perform mixed precision quantization: Quantize the weight tensors of each layer according to the allocated bit width (e.g., convert from 32-bit floating-point to a specified integer bit width), and finally generate lightweight network parameters compressed through the three stages of "structured pruning - dynamic sparsity - mixed precision quantization".

[0029] In a preferred embodiment of the present invention, step 2 above may include: Step 2.1: In response to the deployment command, load the pruned and quantized network parameters into the heterogeneous memory architecture of the edge computing device; Step 2.2: Based on the loaded network parameters, the topology of the model computation graph is parsed by the heterogeneous computing scheduler. The feature extraction operator is assigned to the neural network processor according to the parsing result, and the classification operator is offloaded to the multi-core central processing unit according to the parsing result. Step 2.3: For the feature extraction operators allocated to the neural network processor, the access frequency of the model parameters associated with the operators is analyzed according to the virtual memory paging mechanism, and the high-frequency parameter pages are dynamically cached to the on-chip static memory based on the access frequency analysis results. Step 2.4: After the neural network processor completes the execution of the feature extraction operator, it transmits the generated feature tensor directly to the memory space of the central processing unit through the direct memory access controller, according to the state of the cached data pages, via physical address mapping. Step 2.5: After receiving the transmitted feature tensor, the central processing unit executes the unloaded classification operator to generate the inference result tensor, and outputs the inference result tensor through the edge device.

[0030] In this embodiment of the invention, the execution efficiency and real-time performance of lightweight deployment of large AI models are significantly improved through the collaborative scheduling and efficient management of heterogeneous resources of edge devices: loading the pruned and quantized parameters onto a heterogeneous memory architecture can adapt to the diverse storage structures of edge devices and improve the flexibility of parameter access; allocating feature extraction operators to neural network processors (NPUs) and offloading classification operators to multi-core central processing units (CPUs) through a heterogeneous computing scheduler can fully leverage the computing advantages of different hardware (NPUs are good at parallel feature extraction, and CPUs are good at logical classification), thereby improving computing power utilization; caching high-frequency parameters to on-chip static memory based on a virtual memory paging mechanism can reduce memory access latency and accelerate operator execution; using a direct memory access controller (DMA) to achieve zero-copy transfer of feature tensors avoids the time-consuming data copying between processors and shortens the data flow cycle; finally, the CPU executes the classification operator to output the inference result, forming an efficient closed loop of "parameter loading - operator allocation - memory optimization - data transmission - inference output". The overall process, through the synergy of hardware feature adaptation, precise resource scheduling, and data transmission optimization, significantly reduces inference latency under the condition of limited edge device resources, ensuring the realization of real-time response requirements in complex scenarios (such as industrial equipment fault diagnosis).

[0031] In this embodiment of the invention, the specific steps include: Step 2.1: Respond to deployment instructions and load the pruned and quantized network parameters into the heterogeneous memory architecture of the edge computing device. When a startup or scheduling instruction is received from the deployment management system, the parameter loading process is triggered. The integrity of the pruned and quantized network parameters (which exist in the form of binary files or serialized data) is verified (e.g., hash value comparison). The parameters that pass the verification are loaded into the heterogeneous memory architecture of the edge device (including on-chip static RAM, dynamic random access memory, DRAM, etc.) through the memory controller. The parameters are stored hierarchically according to their storage characteristics. Core parameters expected to be accessed frequently (such as feature extraction weights) are preferentially mapped to low-latency SRAM, while large-capacity general parameters are mapped to DRAM. An association mapping between logical addresses and physical storage units is established to ensure that the parameters can be accessed efficiently.

[0032] Step 2.2: Based on the loaded network parameters, the computation graph topology is parsed using a heterogeneous computing scheduler. Based on the parsing results, feature extraction operators are allocated to the neural network processors, and classification operators are offloaded to the multi-core CPU based on the parsing results. The heterogeneous computing scheduler calls the graph parsing interface to parse the topology of the computation graph and obtain the type of each operator (such as convolution, fully connected), data dependencies, total computation (number of floating-point operations), and input data volume (number of bytes of feature tensors). Calculate the computational intensity of each operator: Calculate using the formula "Operator computational intensity = Total operator computation ÷ Operator input data volume", and set a computational intensity threshold (e.g., 1000 FLOPS / byte); Allocate operators based on computational intensity: Assign feature extraction operators (e.g., convolution, attention operators, computationally intensive) with computational intensity higher than the threshold to the Neural Processing Unit (NPU), and offload classification operators (e.g., fully connected, softmax operators, data intensive) with computational intensity lower than the threshold to the multi-core Central Processing Unit (CPU); Monitor hardware load rate in real time: Use the formula "Hardware load rate = (Currently allocated operator computation ÷ Maximum hardware computation capacity) × 100%" to ensure that the load rate of the NPU and CPU is maintained within a reasonable range of 30%-70%, avoiding overload or resource idleness.

[0033] Step 2.3: For the feature extraction operators allocated to the neural network processor, perform access frequency analysis on the model parameters associated with the operators according to the virtual memory paging mechanism, and dynamically cache high-frequency parameter pages to on-chip static memory based on the access frequency analysis results. Based on the virtual memory paging mechanism, the model parameters associated with the NPU are divided into several parameter pages of a fixed size (e.g., 4KB / page), and the memory controller records the number of accesses to each parameter page within a unit of time (e.g., 1 second). Calculate the parameter page access frequency: Calculate using the formula "Parameter page access frequency = Total number of accesses per unit time ÷ Statistical time", and set a frequency threshold (e.g., 5 times / second). Dynamic caching of high-frequency parameter pages: Parameter pages accessed more than a threshold are marked as "hot pages" and migrated to on-chip static memory (SRAM) cache using the LRU (Least Recently Used) page replacement algorithm, while low-frequency parameter pages are kept in DRAM; Evaluate caching effectiveness: Monitor according to the formula "Memory cache hit rate = (Number of accesses that hit SRAM ÷ Total number of accesses) × 100%" to ensure that the hit rate is ≥80% in order to reduce memory access latency.

[0034] Step 2.4: After the neural network processor completes the execution of the feature extraction operator, it directly transfers the generated feature tensor to the central processing unit's memory space via physical address mapping, according to the cached data page state. After the NPU completes feature extraction, it outputs a feature tensor and notifies the system via an interrupt signal. The system checks the status of the data pages in SRAM (such as whether there are updates that need to be synchronized), and triggers the transmission process after confirming that no additional processing is required. Configure the Direct Memory Access Controller (DMA): Specify the source address (NPU memory physical address), destination address (CPU DRAM pre-allocated area physical address), and total data volume (bytes) of the feature tensor; Zero-copy transfer is performed: The DMA controller transfers data directly through physical address mapping without CPU relay. The time consumption is evaluated according to the formula "DMA transfer time = total data volume of feature tensor ÷ actual DMA transfer bandwidth", ensuring that the transfer time accounts for ≤20% of the total inference latency.

[0035] Step 2.5: After receiving the transmitted feature tensor, the central processing unit executes the unloaded classification operator to generate the inference result tensor, and outputs the inference result tensor through the edge device. The CPU completes the interrupt-aware transfer via DMA and reads the feature tensor data from DRAM. Call the unloaded classification operators (such as fully connected layers and softmax layers), perform calculations with the feature tensor as input, and generate an inference result tensor containing inference results (such as device fault type and confidence level); The inference result tensor is sent to the target terminal (such as a factory monitoring system or alarm device) through the output interface of the edge device (such as an Ethernet port or local bus) to complete the end-to-end inference output.

[0036] In a preferred embodiment of the present invention, step 3 above may include: Step 3.1: After receiving the inference result tensor output by the edge device, detect whether the tensor contains a preset abnormal feature vector; Step 3.2: When an abnormal feature vector is detected, the teacher model is invoked to perform inference based on the input data that triggered the anomaly detection, and the corresponding baseline probability distribution is generated. Step 3.3: Based on the baseline probability distribution, calculate the KL divergence between the edge device output probability distribution and the baseline probability distribution; Step 3.4: Using the KL divergence value as the optimization objective, generate a gradient update instruction and add Laplace noise to this instruction; Step 3.5: Synchronize the noise gradient update command to the edge device and control the edge device to apply the command to update the network parameters.

[0037] In this embodiment of the invention, dynamic optimization and secure updates of the edge model are achieved through an anomaly-driven knowledge transfer and privacy protection mechanism: by detecting preset anomaly feature vectors in the inference results, the edge model's inadequacy in adapting to special scenarios (such as sudden failures of industrial equipment) can be captured in a timely manner, ensuring that anomalies are accurately identified; a benchmark probability distribution is generated by calling the teacher model, providing a high-confidence optimization reference for the edge model and ensuring the reliability of the update direction; the difference between the edge output and the benchmark distribution is quantified by KL divergence value, providing a clear optimization target for gradient updates and ensuring that model parameter adjustments can effectively narrow the performance gap; Laplace noise is added to the gradient update instruction, protecting sensitive data (such as equipment operation details) through differential privacy technology while achieving knowledge transfer, avoiding privacy leaks; finally, the noise-adding instruction is synchronized to the edge device to complete the parameter update, enabling the edge model to dynamically adapt to anomaly scenarios, improving inference accuracy while ensuring data security.

[0038] In this embodiment of the invention, the specific steps include: Step 3.1, Pre-set anomaly feature vector library: Based on historical anomaly data (such as rare fault modes and high-risk feature combinations) of the target task (such as industrial equipment fault diagnosis), analyze and extract anomaly feature vector templates (such as vibration feature values ​​that exceed the normal range and robotic arm posture vectors at specific angles), and set key thresholds for feature vectors (such as numerical range and pattern similarity standards). Anomaly detection: Receive the inference result tensor (containing classification results and feature vectors) output by the edge device, and use a vector similarity matching algorithm (such as cosine similarity) to compare the feature vectors in the tensor with a preset anomaly feature vector library; if the similarity between a feature vector and a template in the library is higher than a set threshold (such as 0.8), it is determined that the tensor contains a preset anomaly feature vector, triggering the subsequent update process.

[0039] Step 3.2, Construction of the teacher model: It needs to be trained on a large amount of labeled data to ensure high accuracy and stability on the target task (such as image classification, anomaly detection, etc.), serving as a "benchmark" for judging whether the inference results of the edge lightweight model are reliable.

[0040] Pre-setting of abnormal feature vectors: Based on task requirements, feature vector templates for triggering anomaly detection are pre-set by analyzing historical abnormal data or defining key feature thresholds (such as feature values ​​that exceed the normal range, feature combinations of specific patterns, etc.).

[0041] Trace the input data: Locate the original input data that triggers anomaly detection (such as time series data from equipment vibration sensors and image frames from industrial cameras), and preprocess the data (such as denoising, standardization, and format conversion) to ensure that it matches the input format of the teacher's model; Generate baseline probability distribution: The teacher model output includes the probability distribution of each target category (e.g., "bearing wear: 95%, normal: 5%)", which serves as a benchmark for measuring the accuracy of the marginal model output.

[0042] Step 3.3, Extract the edge probability distribution: From the inference result tensor of the edge device, parse out the probability distribution of its output (such as "bearing wear: 60%, circuit overheating: 30%, normal: 10%" predicted by the edge model). Calculate KL divergence: Calculate it according to the formula "KL divergence value = probability of each category in the marginal probability distribution × (log ratio of the probability of the category in the marginal distribution to the probability in the baseline distribution of the teacher model)". That is, for each category i, calculate "P_edge(i)×ln(P_edge(i) / P_teacher(i))" and then sum them to get the total divergence value. The significance of divergence: The larger the KL divergence value, the more significant the difference between the output of the marginal model and the teacher model, and the gap needs to be narrowed by parameter updates.

[0043] Step 3.4, Generate gradient update instructions: With the optimization objective of minimizing the KL divergence value calculated in step 3.3, the gradients (i.e., the direction and magnitude of parameter adjustment) of each layer parameter (such as attention head weights and feedforward network bias) of the edge model are calculated through the backpropagation algorithm to form gradient update instructions; Adding Laplace noise: To protect data privacy, noise is generated according to a Laplace distribution. The probability density function of the noise value is... ,in Scale parameters (set according to privacy budget, such as...) =0.1), This is the noise value. It is a natural constant; the generated noise is superimposed on the corresponding parameter adjustment amount of the gradient update instruction to obscure the relevant information of the original data and avoid privacy leakage.

[0044] Step 3.5, Secure Transmission: Transmit the noisy gradient update command to the edge device via an encrypted communication protocol (such as TLS 1.3), along with an integrity check code (such as SHA-256 hash) to prevent the command from being tampered with; Parameter update: The edge device receives the instruction and verifies the check code. After passing the verification, it parses the parameter adjustment information in the instruction, calls the model parameter update interface, and performs incremental updates on the network parameters of the lightweight model (such as the weights and biases after pruning and quantization) according to the adjustment amount. Confirmation feedback: After the update is completed, the edge device returns a parameter update confirmation signal, and the system records logs such as update time and KL divergence change to evaluate the update effect.

[0045] In a preferred embodiment of the present invention, step 4 above may include: Step 4.1: Receive the updated network parameters, synchronously parse the computational load matrix and task dependency data reported in real time by the edge nodes, generate the execution order topology data between operators, and extract the maximum latency threshold of the critical path by combining latency constraints. Step 4.2 transforms the critical path latency threshold into resource dimension constraints and constructs a three-dimensional resource space coordinate system. The computing power axis is normalized according to the proportion of actual available floating-point computing power of the neural network processor and the central processing unit; the memory axis maps the on-chip static memory and dynamic random access memory capacity into a continuous virtual address space based on a hierarchical storage architecture; the network input / output axis uses the product of transmission bandwidth and the reciprocal of end-to-end latency as the quantization benchmark. Step 4.3: Based on the updated network parameters and the computational load matrix, calculate the resource demand vector and project it onto the established three-dimensional space. The computational power demand vector is determined by the peak computational cost of the model operators; the memory demand vector is determined by the sum of the model parameter volume and the peak occupancy of the activation tensor; the network input and output demand vector is determined by dividing the amount of data transmitted by the feature tensor by the critical path delay threshold. Construct a multi-dimensional hypercube with delay constraints in the three-dimensional space to complete the mapping of resources to the constraint space.

[0046] In this embodiment of the invention, by constructing a multi-dimensional resource mapping mechanism with latency constraints, the precise matching of edge node resource requirements and hardware capabilities is achieved, significantly improving the scientific nature and adaptability of resource scheduling in complex scenarios: Based on updated parameters, real-time load matrices, and task-dependent operator topology data, the maximum latency threshold of critical paths is extracted in conjunction with latency constraints, ensuring that the constraints fit the actual operating scenario and avoiding blind resource allocation detached from the load; the latency threshold is transformed into constraints in a three-dimensional resource space (computing power, memory, network I / O), and the resource dimensions are unified through normalization and quantization benchmarks, solving the problem that heterogeneous resources (NPU / CPU computing power, hierarchical storage, transmission bandwidth) are difficult to directly compare, making resource characteristics intuitively measurable; the resource requirement vector is calculated and projected onto a three-dimensional space to construct a hypercube with latency constraints, clearly presenting the correlation between the model's specific requirements for computing power, memory, and network and latency limitations, providing a concrete spatial model for subsequent resource optimization and allocation. The overall process transforms abstract resource requirements and latency constraints into quantifiable and mappable spatial coordinates, realizing the shift from "experience-based scheduling" to "data-driven scheduling," and can adapt to the dynamic load of edge nodes and the characteristics of heterogeneous hardware.

[0047] In this embodiment of the invention, the specific steps include: Step 4.1: Receive the updated network parameters (reflecting the current computational characteristics of the model), and at the same time obtain the computational load matrix (including hardware status data such as available floating-point computing power and load rate of NPU / CPU) and task dependency data (such as a dependency list of "operator A output → operator B input") reported in real time by the edge nodes. Based on task dependency data, for operator dependency relationships (in a directed acyclic graph, an edge represents the dependency of "preceding operator → subsequent operator", that is, the output of the preceding operator is the input of the subsequent operator): count the "in-degree" of each operator (node): which refers to the number of other operators that the operator depends on (that is, the total number of edges pointing to the operator, for example, if operator B depends on operators A and C, then the in-degree of B is 2).

[0048] Construct an "adjacency list": Record the "subsequent operators" of each operator (i.e., the operators that the output of the operator points to; for example, if the output of A is the input of B and D, then the adjacency list of A contains B and D).

[0049] Find all operators with an in-degree of 0 (i.e., operators that do not depend on any other operators and can be used as the starting point for execution), and add these operators to a queue. Initialize an empty "topology sequence" (to store the execution order of operators). When the queue is not empty, take an operator from the queue (denoted as the current operator) and add it to the topology sequence (indicating that the operator is executable). Iterate through all subsequent operators of the current operator in the adjacency list. For each subsequent operator: decrement its in-degree by 1 (because the current operator has been executed, one of the subsequent operator's dependencies has been met). If the in-degree of the subsequent operator becomes 0 (i.e., all its dependent operators have been executed), add it to the queue.

[0050] Combined with the preset end-to-end total latency constraint (e.g., 50ms in industrial scenarios), the maximum allowable latency of the critical path is calculated according to the formula "critical path maximum latency threshold = end-to-end total latency constraint × critical path proportion coefficient (preset, e.g., 0.6)" (e.g., 50ms × 0.6 = 30ms). The longest execution path (critical path) from input to output is identified in the topology graph, and its maximum allowable latency is set to this threshold.

[0051] Step 4.2: Transform the maximum critical path latency threshold (e.g., 30ms) into resource-level constraints (e.g., resource allocation must satisfy critical path latency ≤ 30ms). Construct a three-dimensional resource space coordinate system: Computing power axis: The actual available floating-point computing power of edge node NPU and CPU is statistically analyzed (e.g., NPU can use 100 GFLOPS, CPU can use 20 GFLOPS, total available 120 GFLOPS). It is calculated according to the formula "normalized value of a certain hardware = available computing power of the hardware ÷ total available computing power" (NPU: 100 ÷ 120 ≈ 0.83, CPU: 20 ÷ 120 ≈ 0.17). The axis range is mapped to 0-1. Memory axis: Statistical hierarchical storage capacity (e.g., SRAM 1MB + DRAM 100MB, total 101MB), calculated according to the formula "Mapping value of a certain storage cell = Capacity of that cell ÷ Total storage capacity" (SRAM: 1 ÷ 101 ≈ 0.01, DRAM: 100 ÷ 101 ≈ 0.99), the axis range is mapped to a continuous virtual address space of 0-1; Network I / O axis: Calculated using the formula "Quantization value = Transmission bandwidth (bytes / second) × Reciprocal of end-to-end latency (1 / second)" (e.g., bandwidth 100MB / s, latency 30ms = 0.03s, Quantization value = 100 × (1 ÷ 0.03) ≈ 3330MB / s). 2 Then normalize it to the 0-1 range to serve as a quantitative benchmark for network transmission capacity.

[0052] Step 4.3, Computational power demand vector: Based on the updated network parameters, calculate the peak computational cost of all operators (e.g., 50 GFLOPS for convolution operator, 5 GFLOPS for fully connected operator), and obtain 55 GFLOPS using the formula "Computational power demand vector value = sum of peak computational cost of each operator"; Memory demand vector: Calculate using the formula "Memory demand vector value = model parameter volume + peak activation tensor usage" (e.g., 20MB for parameters + 30MB for peak activation tensor = 50MB); Network I / O demand vector: Calculate the total data transmission volume of feature tensors on the critical path (e.g., 10MB), and obtain the calculation using the formula "Network I / O demand vector value = data transmission volume ÷ maximum latency threshold of critical path" (10MB ÷ 0.03s ≈ 333.3MB / s). Spatial projection and hypercube construction: Project the three demand vector values ​​mentioned above onto the three-dimensional resource space according to the normalization rule in step 4.2 to obtain the coordinates of the demand point; with this point as the center, and combined with the upper and lower limits of resources corresponding to the critical path latency threshold (such as computing power demand ≤ 80% of the total available computing power), construct a multi-dimensional hypercube with latency constraints in the three-dimensional space (the boundaries of each dimension are the minimum / maximum of the resource demand), and complete the mapping from the actual resource demand to the constraint space.

[0053] In a preferred embodiment of the present invention, step 4 above may include: Step 4.4: Using the multidimensional constraint space as input, recursively perform spatial bisection partitioning along the direction of the normal vector of the time delay constraint hyperplane, calculate the variance of each resource dimension, and select the dimension with the largest variance as the splitting axis; use the median of the resource vectors in the current partition as the cutting point. Step 4.5: Perform latency compliance verification on the leaf partitions generated by recursive partitioning, mark the partitions that meet the latency threshold as feasible subspaces, construct a locality-sensitive hash index table within the feasible subspaces, and use the linear combination of resource utilization and the reciprocal of latency as the hash key; bucket the vectors according to the Pareto front distance. Step 4.6: Input the real-time resource demand vector into the hash index table, perform an approximate nearest neighbor search, minimize the weighted sum of resource idle rate and latency margin; extract the non-dominated solution set from the same hash bucket, sort the candidate solution set by Pareto order, and select the highest-order solution as the final resource allocation scheme.

[0054] In this embodiment of the invention, efficient spatial partitioning, precise feasible subspace filtering, and intelligent final solution search are employed: Multidimensional space is recursively partitioned along the normal vector of the latency constraint hyperplane, selecting the dimension with the largest variance as the dividing axis and cutting with the median. This allows focusing on the dimension with the most significant differences in resource distribution, achieving refined spatial partitioning and reducing invalid search ranges. Latency compliance verification is performed on leaf partitions, retaining only feasible subspaces that meet the threshold, ensuring subsequent searches focus on effective solutions. Simultaneously, locality-sensitive hash indexing and Pareto front distance binning are used to correlate resource utilization with latency characteristics, significantly shortening search time. Nearest neighbor search is used to quickly match real-time requirements, minimizing the weighted sum of resource idle rate and latency margin, combined with Pareto ranking of non-dominated solution sets. The overall process balances the accuracy of spatial partitioning, quickly adapting to the dynamic load and heterogeneous resource characteristics of edge nodes, providing a balanced solution for real-time services (such as industrial equipment fault diagnosis) that balances low latency and high resource utilization.

[0055] In this embodiment of the invention, the specific steps include: Step 4.4: Using the multidimensional constraint space as input, recursively perform spatial bisection partitioning along the direction of the delay constraint hyperplane normal vector: Calculate the variance of each resource dimension (such as computing power, memory, network I / O). The calculation method is "variance of a resource dimension = sum of the squares of the differences between all resource vector values ​​in that dimension and the mean of that dimension, divided by the number of vectors". Select the dimension with the largest variance as the splitting axis. When determining the cut point, calculate the median of the resource dimension corresponding to the splitting axis in the current partition. The calculation method is "sort all resource vector values ​​of that dimension in the current partition in ascending order and take the value of the middle position (if the number of vectors is even, take the average of the two middle values)". Use this median as the cut point for spatial partitioning.

[0056] Step 4.5: Perform latency compliance verification on the leaf partitions generated by recursive partitioning, and mark the partitions that meet the latency threshold as feasible subspace sets; construct a local sensitive hash index table within the feasible subspace sets: where the hash key is a linear combination of resource utilization and the reciprocal of latency, resource utilization is calculated as "the amount of resources allocated in a certain dimension ÷ the total amount of available resources in that dimension", and the reciprocal of latency is calculated as "1 ÷ the actual execution latency (seconds) of the resource allocation scheme", and the hash key is specifically "(resource utilization × weight 1) + (reciprocal of latency × weight 2)" (weight 1 and weight 2 are preset priority coefficients, set according to scenario requirements, and the sum is usually 1); bucket the vectors according to the Pareto front distance, which is calculated as "the square root of the sum of the squares of the differences between the resource vector and the nearest point on the Pareto front in each resource dimension".

[0057] Step 4.6: Input the real-time resource demand vector into the hash index table and perform an approximate nearest neighbor search. The search process needs to minimize the weighted sum of resource idle rate and latency margin: where the resource idle rate is calculated as "1 - resource utilization rate", and the latency margin is calculated as "actual latency of the resource allocation scheme - maximum latency threshold of the critical path (if the result is negative, take 0)". The objective function value is "(resource idle rate × weight 3) + (latency margin × weight 4)" (weights 3 and 4 are dynamically adjusted according to the resource tension, and the sum is usually 1). Extract the non-dominated solution set from the same hash bucket, sort the candidate solution set by Pareto order, and select the highest-ranked solution as the final resource allocation scheme.

[0058] In a preferred embodiment of the present invention, step 5 above may include: Step 5.1: Receive the Pareto final resource allocation scheme and parse the computing power allocation ratio, memory space division and network input / output bandwidth quota in the scheme to generate parsed resource allocation data; Step 5.2: Based on the resource allocation data, calculate the runtime constraints of the instance, allocate the neural network processor computing power quota and the number of on-chip static memory page frames to the feature extraction operator instance; allocate the central processing unit core binding strategy and dynamic random access memory limit to the classification operator instance; and set the inter-container communication bandwidth threshold based on the quantization value of the network input and output axes. Step 5.3: Based on the set runtime constraints, dynamically monitor the end-to-end inference latency of edge devices; when the real-time latency deviates from the service level agreement threshold, trigger the elastic scaling operation of the container instance; Step 5.4: Based on the elastic scaling operation, the updated network parameters are injected in real time through the container runtime interface to ensure the consistency of the model state during elastic scaling, thereby keeping the end-to-end inference latency within the service level agreement threshold range.

[0059] In this embodiment of the invention, by implementing and dynamically optimizing the Pareto final resource allocation scheme, precise scheduling of edge device resources and stable control of inference latency are achieved: The computing power, memory, and network quotas in the final scheme are analyzed to provide a clear basis for resource allocation, ensuring that the allocation direction is consistent with the final solution; based on the analyzed data, suitable hardware resources (NPU computing power, CPU core binding, storage page frames, etc.) and communication bandwidth thresholds are configured for feature extraction operators and classification operators respectively, achieving precise matching between operators and hardware characteristics and improving resource utilization; end-to-end inference latency is dynamically monitored, and elastic scaling is triggered when it deviates from the Service Level Agreement (SLA) threshold, which can quickly respond to load fluctuations (such as sudden data volume growth) and avoid latency exceeding the standard; updated parameters are injected during scaling through the container runtime interface to ensure model state consistency and prevent fluctuations in inference accuracy or latency during scaling. The overall process forms a closed loop of "final solution implementation - dynamic monitoring - flexible adjustment - stable status", which can not only give full play to the efficiency of the final resource allocation, but also respond to environmental changes through dynamic adaptation, and ultimately ensure that the end-to-end inference latency remains stable within the SLA threshold.

[0060] In this embodiment of the invention, the specific steps include: Step 5.1: Receive the output Pareto optimal resource allocation scheme and extract the core allocation parameters through the scheme parsing interface: Computing power allocation ratio: e.g., NPU computing power accounts for 70%, CPU computing power accounts for 30% (based on the total available floating-point computing power of edge nodes); Memory space allocation: e.g., 20% of the capacity of on-chip static RAM (SRAM) is allocated to feature extraction operators, and 60% of the capacity of dynamic random access memory (DRAM) is allocated to classification operators; Network input / output bandwidth quota: e.g., the communication bandwidth quota between containers (NPU containers and CPU containers) is 50% of the total available bandwidth; The above parameters are converted into structured data (such as JSON format) to generate parsed resource allocation data, which serves as the basis for subsequent resource calculation and allocation.

[0061] Step 5.2: Calculate the NPU computing power quota using the formula "NPU computing power quota = total available NPU computing power of edge nodes × NPU computing power allocation ratio". For example, if the total available NPU computing power is 100 GFLOPS and the allocation ratio is 70%, then the quota = 100 × 70% = 70 GFLOPS. Calculate the SRAM page frame count using the formula "SRAM page frame count = total SRAM page frame count × SRAM space allocation ratio". For example, if the total SRAM page frames are 1024 pages and the allocation ratio is 20%, then the page frame count = 1024 × 20% ≈ 205 pages, which are used to cache high-frequency parameters. Classification operator instance resource configuration: Configure core binding rules according to CPU core allocation strategy (e.g., binding physical cores 1-3) to avoid core switching overhead; Set memory usage threshold according to DRAM allocation ratio (e.g., 60%) (exceeding this threshold will trigger OOM protection). The threshold for inter-container communication bandwidth is calculated using the formula "Total available network bandwidth of edge nodes × Network input / output bandwidth quota ratio". For example, if the total available bandwidth is 100MB / s and the quota ratio is 50%, then the threshold is 100 × 50% = 50MB / s, ensuring that the transmission does not exceed the limit.

[0062] Step 5.3, Real-time latency calculation: Deploy the latency monitoring module and accumulate the time of key nodes according to the formula "Real-time end-to-end latency = Feature extraction time + Data transmission time + Classification inference time" (e.g., feature extraction 20ms + transmission 10ms + classification 15ms = 45ms). SLA threshold deviation judgment: Calculate according to the formula "upper limit of delay threshold = SLA threshold × (1 + preset deviation ratio, such as 10%)", such as SLA threshold 50ms, then upper limit = 50 × 1.1 = 55ms; calculate according to the formula "lower limit of delay threshold = SLA threshold × (1 - preset deviation ratio, such as 30%)", such as lower limit = 50 × 0.7 = 35ms; Trigger scaling up / down: When the real-time latency exceeds the upper limit for 3 consecutive times (e.g., >55ms), increase the number of NPU / CPU instances (e.g., increase the number of NPU instances from 1 to 2); if it is below the lower limit (e.g., <35ms), reduce the number of redundant instances (e.g., reduce the number of NPU instances from 2 to 1).

[0063] Step 5.4, parameter synchronization: Load the optimized network parameters from step 3.5 into the newly started container instance through the container runtime interface (such as CRI-O), and replace the old parameters in the existing instance through the hot update interface; State consistency verification: Calculate using the formula "Model state consistency error rate = |Inference result after scaling down - Inference result before scaling down| ÷ Inference result before scaling down×100%". For example, if the result before scaling down is 90% and after scaling down is 89.5%, the error rate = |89.5% - 90%| ÷ 90%×100% ≈ 0.56%, ensuring the error is <1%. Closed-loop optimization: Continuously monitor the end-to-end latency after scaling up or down. If it still deviates from the SLA threshold (e.g., >55ms or <35ms), repeat steps 5.3-5.4 to adjust until the latency stabilizes within the range of 45-55ms.

[0064] like Figure 2 As shown, embodiments of the present invention also provide a lightweight deployment system for large AI models in complex scenarios, including: The acquisition module is used to perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head, dynamically sparsify the activation state of the feedforward network according to the input tensor entropy value, and use dynamic mixed precision quantization to obtain the pruned and quantized network parameters. The inference module is used to deploy the pruned and quantized network parameters to edge computing devices, allocate feature extraction operators to neural network processors through a heterogeneous computing scheduler, and offload classification operators to multi-core central processing units; it manages on-chip memory using a virtual memory paging mechanism, achieves zero-copy data transfer using a direct memory access controller, and outputs inference result tensors. The computation module is used to perform differential privacy gradient synchronous updates when the inference result tensor is detected to contain a preset abnormal feature vector. This is done by minimizing the difference in probability distribution between the edge device output and the teacher model output to transfer knowledge and obtain updated network parameters. Based on the updated network parameters, the real-time computation load matrix, task-dependent data, and time delay constraints, the computing power, memory, and network input / output resources of the edge nodes are mapped into a multi-dimensional constraint space. A multi-dimensional tree index is recursively constructed to locate partitions that meet the time delay requirements, and an approximate nearest neighbor search is used to match the Pareto final resource allocation. The processing module is used to dynamically allocate computing instance resources based on the final resource allocation ratio, and maintain end-to-end latency within the service level agreement threshold.

[0065] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A lightweight deployment method for large AI models in complex scenarios, characterized in that, The method includes: Step 1: Perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head, dynamically sparsify the activation state of the feedforward network according to the input tensor entropy value, and use dynamic mixed precision quantization to obtain the pruned and quantized network parameters. Step 2: Deploy the pruned and quantized network parameters to the edge computing device. Use the heterogeneous computing scheduler to allocate the feature extraction operator to the neural network processor and the classification operator to the multi-core central processing unit. Combine the virtual memory paging mechanism to manage the on-chip memory and use the direct memory access controller to achieve zero-copy data transmission and output the inference result tensor. Step 3: When the inference result tensor is detected to contain a preset abnormal feature vector, differential privacy gradient synchronous update is performed by minimizing the difference between the probability distribution of the edge device output and the teacher model output to obtain the updated network parameters. Step 4: Based on the updated network parameters, real-time calculation load matrix, task-dependent data and time delay constraints, map the edge node computing power, memory and network input / output resources into a multi-dimensional constraint space; recursively construct a multi-dimensional tree index to locate partitions that meet the time delay, and use an approximate nearest neighbor search to match the Pareto final resource allocation. Step 5: Based on the final resource allocation, dynamically allocate computing instance resources to maintain end-to-end latency within the service level agreement threshold.

2. The lightweight deployment method for large AI models in complex scenarios according to claim 1, characterized in that, Structured pruning is performed on the pre-trained Transformer network based on the importance score of the attention head. The activation state of the feedforward network is dynamically sparsified according to the input tensor entropy value, and dynamic mixed-precision quantization is used to obtain the pruned and quantized network parameters, including: Step 1.1: Perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head to generate the pruned network model; Step 1.2: Input the pruned network model into the feedforward network, and dynamically sparsify the activation state according to the entropy value of the input tensor to generate a feedforward sub-network with sparse activation mode. Step 1.3: Perform dynamic mixed-precision quantization on the weight tensors of the feedforward subnetwork with sparse activation mode, adaptively allocate bit width based on layer sensitivity analysis, and generate lightweight network parameters with three-stage compression.

3. The lightweight deployment method for large AI models in complex scenarios according to claim 2, characterized in that, The pruned and quantized network parameters are deployed to edge computing devices, and the feature extraction operators are allocated to neural network processors through a heterogeneous computing scheduler, while the classification operators are offloaded to multi-core central processing units. By combining virtual memory paging mechanisms to manage on-chip memory and utilizing a direct memory access controller to achieve zero-copy data transfer, the inference result tensor is output, including: Step 2.1: In response to the deployment command, load the pruned and quantized network parameters into the heterogeneous memory architecture of the edge computing device; Step 2.2: Based on the loaded network parameters, the topology of the model computation graph is parsed by the heterogeneous computing scheduler. The feature extraction operator is assigned to the neural network processor according to the parsing result, and the classification operator is offloaded to the multi-core central processing unit according to the parsing result. Step 2.3: For the feature extraction operators allocated to the neural network processor, the access frequency of the model parameters associated with the operators is analyzed according to the virtual memory paging mechanism, and the high-frequency parameter pages are dynamically cached to the on-chip static memory based on the access frequency analysis results. Step 2.4: After the neural network processor completes the execution of the feature extraction operator, it transmits the generated feature tensor directly to the memory space of the central processing unit through the direct memory access controller, according to the state of the cached data pages, via physical address mapping. Step 2.5: After receiving the transmitted feature tensor, the central processing unit executes the unloaded classification operator to generate the inference result tensor, and outputs the inference result tensor through the edge device.

4. The lightweight deployment method for large AI models in complex scenarios according to claim 3, characterized in that, When the inference result tensor is detected to contain a preset abnormal feature vector, differential privacy gradient synchronous update is performed by minimizing the difference in probability distribution between the edge device output and the teacher model output, resulting in updated network parameters, including: Step 3.1: After receiving the inference result tensor output by the edge device, detect whether the tensor contains a preset abnormal feature vector; Step 3.2: When an abnormal feature vector is detected, the teacher model is invoked to perform inference based on the input data that triggered the anomaly detection, and the corresponding baseline probability distribution is generated. Step 3.3: Based on the baseline probability distribution, calculate the KL divergence between the edge device output probability distribution and the baseline probability distribution; Step 3.4: Using the KL divergence value as the optimization objective, generate a gradient update instruction and add Laplace noise to this instruction; Step 3.5: Synchronize the noise gradient update command to the edge device and control the edge device to apply the command to update the network parameters.

5. The lightweight deployment method for large AI models in complex scenarios according to claim 4, characterized in that, Based on the updated network parameters, real-time computation load matrix, task-dependent data, and time-latency constraints, the computing power, memory, and network input / output resources of edge nodes are mapped into a multi-dimensional constraint space, including: Step 4.1: Receive the updated network parameters, synchronously parse the computational load matrix and task dependency data reported in real time by the edge nodes, generate the execution order topology data between operators, and extract the maximum latency threshold of the critical path by combining latency constraints. Step 4.2 transforms the critical path latency threshold into resource dimension constraints and constructs a three-dimensional resource space coordinate system. The computing power axis is normalized according to the proportion of actual available floating-point computing power of the neural network processor and the central processing unit; the memory axis maps the on-chip static memory and dynamic random access memory capacity into a continuous virtual address space based on a hierarchical storage architecture; the network input / output axis uses the product of transmission bandwidth and the reciprocal of end-to-end latency as the quantization benchmark. Step 4.3: Based on the updated network parameters and the computational load matrix, calculate the resource demand vector and project it onto the established three-dimensional space. The computational power demand vector is determined by the peak computational cost of the model operators; the memory demand vector is determined by the sum of the model parameter volume and the peak occupancy of the activation tensor; the network input and output demand vector is determined by dividing the amount of data transmitted by the feature tensor by the critical path delay threshold. Construct a multi-dimensional hypercube with delay constraints in the three-dimensional space to complete the mapping of resources to the constraint space.

6. The lightweight deployment method for large AI models in complex scenarios according to claim 5, characterized in that, Recursively construct a multidimensional tree index to locate partitions that meet latency requirements, and use an approximate nearest neighbor search to match the Pareto final resource allocation, including: Step 4.4: Using the multidimensional constraint space as input, recursively perform spatial bisection partitioning along the direction of the normal vector of the time delay constraint hyperplane, calculate the variance of each resource dimension, and select the dimension with the largest variance as the splitting axis; use the median of the resource vectors in the current partition as the cutting point. Step 4.5: Perform latency compliance verification on the leaf partitions generated by recursive partitioning, mark the partitions that meet the latency threshold as feasible subspaces, construct a locality-sensitive hash index table within the feasible subspaces, and use the linear combination of resource utilization and the reciprocal of latency as the hash key; bucket the vectors according to the Pareto front distance. Step 4.6: Input the real-time resource demand vector into the hash index table, perform an approximate nearest neighbor search, minimize the weighted sum of resource idle rate and latency margin; extract the non-dominated solution set from the same hash bucket, sort the candidate solution set by Pareto order, and select the highest-order solution as the final resource allocation scheme.

7. The lightweight deployment method for large AI models in complex scenarios according to claim 6, characterized in that, Based on the final resource allocation, computing instance resources are dynamically allocated to maintain end-to-end latency within the service level agreement threshold, including: Step 5.1: Receive the Pareto final resource allocation scheme and parse the computing power allocation ratio, memory space division and network input / output bandwidth quota in the scheme to generate parsed resource allocation data; Step 5.2: Based on the resource allocation data, calculate the runtime constraints of the instance, allocate the neural network processor computing power quota and the number of on-chip static memory page frames to the feature extraction operator instance; allocate the central processing unit core binding strategy and dynamic random access memory limit to the classification operator instance; and set the inter-container communication bandwidth threshold based on the quantization value of the network input and output axes. Step 5.3: Based on the set runtime constraints, dynamically monitor the end-to-end inference latency of edge devices; when the real-time latency deviates from the service level agreement threshold, trigger the elastic scaling operation of the container instance; Step 5.4: Based on the elastic scaling operation, the updated network parameters are injected in real time through the container runtime interface to ensure the consistency of the model state during elastic scaling, thereby keeping the end-to-end inference latency within the service level agreement threshold range.

8. A lightweight deployment system for large AI models in complex scenarios, wherein the system implements the method as described in any one of claims 1 to 7, characterized in that, include: The acquisition module is used to perform structured pruning on the pre-trained Transformer network based on the importance score of the attention head, dynamically sparsify the activation state of the feedforward network according to the input tensor entropy value, and use dynamic mixed precision quantization to obtain the pruned and quantized network parameters. The inference module is used to deploy the pruned and quantized network parameters to edge computing devices, and to allocate feature extraction operators to neural network processors and classification operators to multi-core central processing units through a heterogeneous computing scheduler. The on-chip memory is managed by combining virtual memory paging mechanism, and zero-copy data transfer is achieved by using direct memory access controller to output inference result tensor. The computation module is used to perform differential privacy gradient synchronous updates when the inference result tensor is detected to contain a preset abnormal feature vector. This is done by minimizing the difference in probability distribution between the edge device output and the teacher model output to transfer knowledge and obtain updated network parameters. Based on the updated network parameters, the real-time computation load matrix, task-dependent data and time delay constraints, the computing power, memory and network input and output resources of the edge nodes are mapped into a multi-dimensional constraint space. Recursively construct a multidimensional tree index to locate partitions that meet the latency requirements, and use an approximate nearest neighbor search to match the Pareto final resource allocation. The processing module is used to dynamically allocate computing instance resources based on the final resource allocation ratio, and maintain end-to-end latency within the service level agreement threshold.

9. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images