Lightweight deployment method of ai large model for complex scene

Abstract

The application provides an AI large model lightweight deployment method for complex scenes, relates to the technical field of edge computing, and comprises the following steps: performing structural pruning on a pre-trained Transformer network based on attention head importance scores, dynamically sparsifying feedforward network activation states according to input tensor entropy values, adopting dynamic mixed precision quantization to obtain pruned and quantized network parameters; deploying the pruned and quantized network parameters to an edge computing device, distributing feature extraction operators to a neural network processor through a heterogeneous computing scheduler, and offloading classification operators to a multi-core central processing unit; combining a virtual memory paging mechanism to manage on-chip storage, using a direct memory access controller to realize zero-copy data transmission, and outputting inference result tensors. The application realizes efficient and reliable operation of a large model on a resource-limited edge node.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing, in particular to an AI large model lightweight deployment method for complex scenarios. BACKGROUND

[0002] With the deep integration of industrial Internet of Things and intelligent manufacturing, large intelligent factories have entered a new stage of multi-device collaboration and full-process digitization. The stable operation of thousands of heterogeneous devices in the factory directly determines the production efficiency, and real-time diagnosis and early warning of device faults have become the core link to ensure the continuous operation of the production line. Currently, fault diagnosis mainly relies on the analysis of multi-source heterogeneous data, including mechanical vibration signals collected by vibration sensors, component appearance images captured by high-definition industrial cameras, and device operating temperatures recorded by temperature sensors, which require AI models to achieve multi-modal feature fusion and accurate identification.

[0003] In the prior art, the lightweight deployment method for large models mainly focuses on model compression and edge computing adaptation, but there are still significant limitations in complex industrial scenarios: the scene adaptability of model compression strategies is insufficient: traditional structured pruning is mainly based on the feature importance score of general data sets, without fully considering the "edge cases" of industrial device faults (such as low-probability but high-risk fault modes, such as abnormal vibration of a robot arm at a specific angle), which can easily lead to the mis-pruning of network structures corresponding to such key features, resulting in an increase in fault detection rate under extreme conditions. SUMMARY

[0004] The technical problem to be solved by the present application is to provide an AI large model lightweight deployment method for complex scenarios, which realizes efficient and reliable operation of large models on resource-constrained edge nodes.

[0005] To solve the above technical problems, the technical solution of the present application is as follows:

[0006] In a first aspect, the AI large model lightweight deployment method for complex scenarios comprises:

[0007] Step 1: Perform structured pruning on the pre-trained Transformer network based on attention head importance score, dynamically sparsify the feedforward network activation state according to the input tensor entropy value, and use dynamic mixed precision quantization to obtain the pruned and quantized network parameters;

[0008] Step 2: Deploy the pruned and quantized network parameters to edge computing devices, distribute feature extraction operators to neural network processors through a heterogeneous computing scheduler, and offload classification operators to multi-core central processing units; manage on-chip memory in combination with a virtual memory paging mechanism, use a direct memory access controller to realize zero-copy data transmission, and output inference result tensors;

[0009] Step 3, when detecting that the inference result tensor contains a preset abnormal feature vector, migrating knowledge by minimizing the difference between the edge device output and the teacher model output probability distribution, performing differential privacy gradient synchronization update, and obtaining updated network parameters;

[0010] Step 4, based on the updated network parameters, real-time calculation of the load matrix, task dependency data and delay constraints, mapping edge node computing power, memory and network input and output resources into a multi-dimensional constraint space; recursively constructing a multi-dimensional tree index to locate a partition that meets the delay, and using an approximate nearest neighbor search to match a Pareto final resource allocation;

[0011] Step 5, based on the final resource allocation, dynamically allocating computing instance resources to maintain end-to-end latency within the service level agreement threshold.

[0012] Further, based on the attention head importance score, the pre-trained Transformer network is structurally pruned, the input tensor entropy value is dynamically sparsified to feed forward network activation state, and dynamic mixed precision quantization is used to obtain pruned and quantized network parameters, including:

[0013] Step 1.1, based on the attention head importance score, the pre-trained Transformer network is structurally pruned to generate a pruned network model;

[0014] Step 1.2, input the pruned network model into the feed forward network, dynamically sparsify the activation state according to the entropy value of the input tensor, and generate a feed forward subnetwork with sparse activation mode;

[0015] Step 1.3, performing dynamic mixed precision quantization on the weight tensor of the feed forward subnetwork with sparse activation mode, adaptively allocating bit width based on layer sensitivity analysis, and generating three-stage compressed lightweight network parameters.

[0016] Further, the pruned and quantized network parameters are deployed to the edge computing device, the feature extraction operator is assigned to the neural network processor through the heterogeneous computing scheduler, and the classification operator is offloaded to the multi-core central processing unit; combined with the virtual memory paging mechanism to manage on-chip memory, and using the direct memory access controller to realize zero-copy data transmission, outputting the inference result tensor, including:

[0017] Step 2.1, in response to the deployment instruction, load the pruned and quantized network parameters into the heterogeneous memory architecture of the edge computing device;

[0018] Step 2.2, based on the loaded network parameters, parse the model computation graph topology through the heterogeneous computing scheduler, and according to the parsing result, assign the feature extraction operator to the neural network processor, and at the same time, according to the parsing result, offload the classification operator to the multi-core central processing unit;

[0019] Step 2.3, according to the virtual memory paging mechanism, the access frequency of the model parameters associated with the feature extraction operator allocated to the neural network processor is analyzed, and based on the analysis result, the high-frequency parameter pages are dynamically cached to the on-chip static memory;

[0020] Step 2.4, after the neural network processor completes the execution of the feature extraction operator, according to the cached data page state, the generated feature tensor is transmitted through the direct memory access controller to the memory space of the central processing unit through physical address mapping;

[0021] Step 2.5, after the central processing unit receives the transmitted feature tensor, the unloaded classification operator is executed to generate an inference result tensor, and the inference result tensor is output through the edge device.

[0022] Further, when it is detected that the inference result tensor contains a preset abnormal feature vector, knowledge is migrated by minimizing the difference between the edge device output and the teacher model output probability distribution, differential privacy gradient synchronization update is performed, and updated network parameters are obtained, including:

[0023] Step 3.1, after receiving the inference result tensor output by the edge device, it is detected whether the tensor contains a preset abnormal feature vector;

[0024] Step 3.2, when the abnormal feature vector is detected, the teacher model is called to execute inference for the input data triggering the abnormal detection, and a corresponding reference probability distribution is generated;

[0025] Step 3.3, based on the reference probability distribution, the KL divergence value of the edge device output probability distribution and the reference probability distribution is calculated;

[0026] Step 3.4, taking the KL divergence value as the optimization target, generating gradient update instructions, and adding Laplace noise to the instructions;

[0027] Step 3.5, synchronizing the noisy gradient update instructions to the edge device to control the edge device to update the network parameters according to the instructions.

[0028] Further, based on the updated network parameters, the real-time calculation load matrix, the task dependency data and the delay constraint, the edge node computing power, the memory and the network input and output resources are mapped to a multi-dimensional constraint space, including:

[0029] Step 4.1, receiving the updated network parameters, synchronously analyzing the calculation load matrix and the task dependency data reported by the edge node in real time, generating operator execution sequence topology data, and extracting the maximum delay threshold of the critical path combined with the delay constraint;

[0030] Step 4.2, convert the critical path latency threshold into resource dimension constraints, build a three-dimensional resource space coordinate system, the computing power axis is normalized according to the actual available floating point computing power ratio of neural network processor and central processing unit; the memory axis is mapped to the continuous virtual address space based on the hierarchical storage architecture; the network input and output axis is to take the product of transmission bandwidth and end-to-end latency as the quantitative benchmark;

[0031] Step 4.3, according to the updated network parameters and the calculation load matrix, calculate the resource demand vector and project it to the established three-dimensional space, the computing power demand vector is determined by the peak calculation amount of model operator; the memory demand vector is determined by the sum of model parameter volume and activation tensor peak occupation; the network input and output demand vector is determined by the feature tensor transmission data divided by the critical path latency threshold, a multi-dimensional hypercube with latency constraint is constructed in the three-dimensional space, and the mapping from resource to constraint space is completed.

[0032] Further, recursively construct a multi-dimensional tree index to locate the partition that meets the latency, and use approximate nearest neighbor search to match the Pareto final resource allocation, including:

[0033] Step 4.4, take the multi-dimensional constraint space as input, recursively perform space bisection along the direction of latency constraint hyperplane normal vector, calculate the variance of each resource dimension, and select the dimension with the largest variance as the partition axis; take the median of the resource vector in the current partition as the cutting point;

[0034] Step 4.5, verify the latency compliance of the leaf partitions generated by recursive partitioning, mark the partitions that meet the latency threshold as feasible subspace set, and in the feasible subspace set, construct a local sensitive hash index table, taking the linear combination of resource utilization and latency inverse as the hash key; bucket the vectors according to the Pareto frontier distance;

[0035] Step 4.6, input the real-time resource demand vector into the hash index table, perform approximate nearest neighbor search, and minimize the weighted sum of resource idle rate and latency margin; extract the non-dominated solution set from the same hash bucket, and sort the candidate solution set according to the Pareto level, and select the highest level solution as the final resource allocation scheme.

[0036] Further, based on the final resource allocation, dynamically allocate computing instance resources to maintain the end-to-end latency within the service level agreement threshold, including:

[0037] Step 5.1, receive the Pareto final resource allocation scheme and parse the computing power allocation ratio, memory space division and network input and output bandwidth quota in the scheme to generate parsed resource allocation data;

[0038] Step 5.2, based on resource allocation data, calculate the runtime constraints of the instance, assign the neural network processor computing power quota and on-chip static memory page frame number to the feature extraction operator instance; assign the central processing unit core binding strategy and dynamic random access memory upper limit to the classification operator instance; and set the inter-container communication bandwidth threshold based on the network input-output axis quantization value;

[0039] Step 5.3, based on the set runtime constraints, dynamically monitor the end-to-end inference delay of the edge device; when the real-time delay deviates from the service level agreement threshold, trigger the elastic scaling operation of the container instance;

[0040] Step 5.4, based on the elastic scaling operation, real-time injection of updated network parameters through the container runtime interface ensures model state consistency during elastic scaling, thereby maintaining the end-to-end inference delay within the service level agreement threshold range.

[0041] The second aspect is an AI large model lightweight deployment system for complex scenarios, which includes:

[0042] The acquisition module is configured to perform structured pruning on the pre-trained Transformer network based on the attention head importance score, dynamically sparsify the feedforward network activation state according to the input tensor entropy value, and obtain the pruned and quantized network parameters by using dynamic mixed precision quantization.

[0043] The inference module is configured to deploy the pruned and quantized network parameters to the edge computing device, assign the feature extraction operator to the neural network processor through the heterogeneous computing scheduler, and offload the classification operator to the multi-core central processing unit; manage the on-chip memory in combination with the virtual memory paging mechanism, and realize zero-copy data transmission using the direct memory access controller to output the inference result tensor.

[0044] The calculation module is configured to, when detecting that the inference result tensor contains a preset abnormal feature vector, migrate knowledge by minimizing the difference between the output of the edge device and the output probability distribution of the teacher model, perform differential privacy gradient synchronization update to obtain updated network parameters; map the edge node computing power, memory and network input-output resources to a multi-dimensional constraint space based on the updated network parameters, real-time calculation load matrix, task dependency data and delay constraints; recursively construct a multi-dimensional tree index to locate a partition that satisfies the delay, and use an approximate nearest neighbor search to match a Pareto final resource allocation.

[0045] The processing module is configured to dynamically allocate computing instance resources based on the final resource allocation to maintain the end-to-end delay within the service level agreement threshold.

[0046] The third aspect is a computing device, which includes:

[0047] One or more processors;

[0048] a memory device storing one or more programs, when executed by the one or more processors, cause the one or more processors to carry out the method.

[0049] In a fourth aspect, a computer-readable storage medium stores a program, which, when executed by a processor, implements the method.

[0050] The above scheme of the present application at least has the following beneficial effects:

[0051] Through multi-stage collaborative design, an efficient solution is provided for lightweight deployment of AI large models in complex scenarios: by combining attention head importance pruning, input tensor entropy dynamic sparsification and dynamic mixed precision quantization, key features are retained while the model is significantly lightened, reducing the risk of losing edge cases; through heterogeneous computing scheduling, virtual memory paging management and zero-copy transmission, the utilization rate of edge device computing power is improved and the inference delay is reduced, meeting real-time requirements; with knowledge transfer based on KL divergence and differential privacy gradient update, dynamic adaptation of edge models to abnormal scenarios is realized on the premise of ensuring data security; through multi-dimensional resource constraint space construction, Pareto final ratio search and container elastic scaling, dynamic response to load fluctuations is realized to ensure that end-to-end latency is stable within the threshold of service level agreement. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flowchart of the AI large model lightweight deployment method for complex scenarios provided by an embodiment of the present application.

[0053] Figure 2 is a schematic diagram of the AI large model lightweight deployment system for complex scenarios provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings; however, they are not limited thereto and can be implemented in various forms. It is to be understood that the embodiments described herein and illustrated in the drawings are provided by way of example only. Additionally, the present disclosure should not be limited by the accompanying drawings, which are merely provided as examples. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0055] As shown in Figure 1 An embodiment of the present application proposes an AI large model lightweight deployment method for complex scenarios, which includes the following steps:

[0056] Step 1, performing structured pruning on the pre-trained Transformer network based on attention head importance score, dynamically sparsifying feedforward network activation state according to input tensor entropy, adopting dynamic mixed precision quantization, and obtaining pruned and quantized network parameters;

[0057] Step 2, deploying the pruned and quantized network parameters to an edge computing device, distributing feature extraction operators to a neural network processor and classification operators to a multi-core central processing unit through a heterogeneous computing scheduler; managing on-chip memory in combination with a virtual memory paging mechanism, and realizing zero-copy data transmission by using a direct memory access controller, and outputting an inference result tensor;

[0058] Step 3, when it is detected that the inference result tensor contains a preset abnormal feature vector, migrating knowledge by minimizing the difference between the output probability distribution of the edge device and the output probability distribution of the teacher model, performing differential privacy gradient synchronous update, and obtaining updated network parameters;

[0059] Step 4, based on the updated network parameters, real-time computing load matrix, task dependency data and delay constraint, mapping edge node computing power, memory and network input and output resources into a multi-dimensional constraint space; recursively constructing a multi-dimensional tree index to locate a partition that meets the delay, and adopting approximate nearest neighbor search to match a Pareto final resource allocation;

[0060] Step 5, based on the final resource allocation, dynamically allocating computing instance resources to maintain end-to-end delay within the threshold of a service level agreement.

[0061] In the embodiment of the present application, through the cooperative compression of structured pruning, dynamic sparsification and mixed precision quantization, the key features are preserved while the model parameter size is greatly reduced, and the edge device resource limitation characteristics are adapted; with the help of heterogeneous computing scheduling, virtual memory management and zero-copy transmission, the edge device computing power utilization rate is improved and the data transmission time is reduced, and the inference real-time performance is guaranteed; through knowledge migration based on probability distribution difference and differential privacy gradient update, the dynamic adaptation of the edge model to abnormal scenes is realized on the premise of protecting data security; combined with multi-dimensional resource constraint space construction and Pareto final allocation search, the edge node computing power, memory and network resources are accurately matched, and the load fluctuation is dynamically responded; finally, through container orchestration and elastic scaling, the end-to-end delay is stabilized within the threshold of the service level agreement, and the efficiency of AI large model edge deployment in complex scenarios is improved.

[0062] In a preferred embodiment of the present application, the above step 1 can include:

[0063] Step 1.1, performing structured pruning on the pre-trained Transformer network based on attention head importance score to generate a pruned network model;

[0064] Step 1.2, input the pruned network model into a feedforward network, dynamically sparsify the activation state according to the entropy value of the input tensor, and generate a feedforward subnetwork with sparse activation mode;

[0065] Step 1.3, performing dynamic mixed precision quantization on the weight tensor of the feedforward subnetwork with sparse activation mode, adaptively allocating bit width based on layer sensitivity analysis, and generating three-stage compressed lightweight network parameters.

[0066] In the embodiments of the present application, through the three-stage collaborative compression strategy, the precise lightweight of the AI large model is realized: the structured pruning based on the importance score of the attention head can selectively retain the network structure that plays a decisive role in key features, avoiding invalid parameter redundancy; combined with the dynamic sparsification of the input tensor entropy value, the calculation intensity can be adaptively adjusted according to the feature importance, reducing redundant calculation while retaining high-entropy key information; and the dynamic mixed precision quantization based on layer sensitivity analysis allocates high bit width to high-sensitive layers and low bit width to low-sensitive layers, maximizing the compression of the weight tensor size while considering the inference accuracy. The three-stage compression mechanism is formed by the cooperation of the three.

[0067] In the embodiments of the present application, the specific steps include:

[0068] Step 1.1, structured pruning stage: by quantifying the importance of each attention head, removing the network components (such as weight matrix, connection layer) corresponding to the low importance head, and simplifying the model structure while retaining the core attention mechanism.

[0069] Activation state sparsification stage: connect the pruned model to the feedforward network, dynamically adjust the neuron activation mode of the feedforward network according to the feature complexity (entropy value) of the input data, and only retain the activation calculation of the high information density area to form a sparsified subnetwork.

[0070] Mixed precision quantization stage: for the weight parameters of the sparsified subnetwork, different quantization bit widths are allocated according to the influence degree (sensitivity) of each layer on the inference accuracy, and high-precision parameters and low-precision parameters are mixed and deployed, finally forming a lightweight model.

[0071] Calculate the importance score of each attention head: for each attention head, calculate the Pearson correlation coefficient between its output feature and the final inference result of the model, and the contribution value to the loss function during training, then weight the two by the preset weight coefficient 1 and weight coefficient 2, and obtain the importance score of the attention head; pruning operation: set the importance score threshold, select the attention heads with scores below the threshold, remove the weight matrices (query, key, value matrices and output projection matrix) and connection layers corresponding to these attention heads, and retain the attention heads with scores above the threshold and their network connections, to generate a pruned network model.

[0072] Step 1.2, calculate the input tensor entropy value: for the input tensor of the feedforward network (output features from the pruned model), count the frequency of each element in the tensor (or the normalized relative value) as the element probability value, calculate the "probability value x the logarithm of the probability value based on 2" for each probability value, and then sum the negative values of all results to obtain the entropy value of the input tensor; determine the sparsification threshold: calculate the threshold according to the formula "sparsification threshold = (average entropy value of the input tensor x proportion coefficient) + minimum retention threshold" (the proportion coefficient and the minimum retention threshold are set according to the scene requirements); dynamic sparsification activation state: divide the local area of the input tensor, calculate the local entropy value of each area, and if the local entropy value is lower than the sparsification threshold, set the activation value of the corresponding feedforward network neuron to zero (or stop the activation calculation); if the local entropy value is higher than the threshold, the activation state is retained, forming a feedforward subnetwork with sparse activation mode.

[0073] Step 1.3, calculate the layer sensitivity score: for each layer of the feedforward subnetwork, respectively use 8-bit quantization and 32-bit floating point calculation for inference, and obtain the accuracy difference (accuracy loss value) of the two, calculate the sensitivity score of the layer according to the formula "sensitivity score = (accuracy loss value ÷ 32-bit floating point inference accuracy) x 100%"; allocate quantization bit width: determine the maximum sensitivity score in all layers, allocate bit width to each layer according to the formula "quantization bit width = base bit width (4 bits) + (sensitivity score of the layer ÷ maximum sensitivity score) x bit width adjustment range (such as 0-12 bits)" (ensure that the bit width of the high sensitivity layer is not lower than the preset minimum value); perform mixed precision quantization: convert the weight tensor of each layer according to the allocated bit width (such as from 32-bit floating point to specified integer bit width), and finally generate the lightweight network parameters compressed by the three-stage compression of "structured pruning-dynamic sparsification-mixed precision quantization".

[0074] In a preferred embodiment of the present application, the above step 2 can include:

[0075] Step 2.1, in response to the deployment instruction, load the pruned and quantized network parameters to the heterogeneous memory architecture of the edge computing device;

[0076] Step 2.2, based on the loaded network parameters, parse the model computation graph topology through the heterogeneous computing scheduler, and according to the parsing result, distribute the feature extraction operators to the neural network processor, and at the same time, according to the parsing result, unload the classification operators to the multi-core central processing unit;

[0077] Step 2.3, for the feature extraction operators distributed to the neural network processor, perform access frequency analysis on the model parameters associated with the operators according to the virtual memory paging mechanism, and based on the access frequency analysis result, dynamically cache the high-frequency parameter pages to the on-chip static memory;

[0078] Step 2.4, when the neural network processor completes the feature extraction operator execution, according to the buffered data page state, the generated feature tensor is transmitted to the memory space of the central processing unit through the direct memory access controller by physical address mapping;

[0079] Step 2.5, after the central processing unit receives the transmitted feature tensor, the offloaded classification operator is executed to generate an inference result tensor, and the inference result tensor is output through the edge device.

[0080] In the embodiment of the present application, through the cooperative scheduling and efficient management of the heterogeneous resources of the edge device, the execution efficiency and real-time performance of the lightweight deployment of the AI large model are significantly improved: the pruned and quantized parameters are loaded into the heterogeneous memory architecture, which can adapt to the diverse storage structures of the edge device and improve the flexibility of parameter access; the feature extraction operator is distributed to the neural network processor (NPU) and the classification operator is offloaded to the multi-core central processing unit (CPU) through the heterogeneous computing scheduler, which can fully utilize the computing advantages of different hardware (NPU is good at parallel feature extraction and CPU is good at logical classification), and improve the utilization rate of computing power; based on the virtual memory paging mechanism, high-frequency parameters are cached to on-chip static storage, which can reduce memory access delay and speed up operator execution; the direct memory access controller (DMA) is used to realize zero-copy transmission of the feature tensor, which avoids the time-consuming copying of data between processors and shortens the data flow cycle; finally, the CPU executes the classification operator to output the inference result, forming an efficient closed loop of "parameter loading-operator distribution-memory optimization-data transmission-inference output". Through the cooperation of hardware characteristics adaptation, resource precise scheduling and data transmission optimization, the inference delay is greatly reduced under the condition of limited resources of the edge device, and the real-time response demand in complex scenarios (such as industrial equipment fault diagnosis) is ensured.

[0081] In the embodiment of the present application, the specific steps include:

[0082] Step 2.1, in response to the deployment instruction, the pruned and quantized network parameters are loaded into the heterogeneous memory architecture of the edge computing device: when receiving the start or scheduling instruction of the deployment management system, the parameter loading process is triggered, and the integrity of the pruned and quantized network parameters (in the form of binary file or serialized data) is checked (such as hash value comparison); the parameters passing the check are loaded into the heterogeneous memory architecture (including on-chip static storage SRAM, dynamic random access memory DRAM, etc.) of the edge device through the memory controller, and are stored in a hierarchical manner according to the storage characteristics; the core parameters expected to be accessed frequently (such as feature extraction weights) are preferentially mapped to the low-latency SRAM, and the large-capacity general parameters are mapped to the DRAM, and the association mapping between the logical address and the physical storage unit is established, so that the parameters can be efficiently accessed.

[0083] Step 2.2, based on the loaded network parameters, the computing graph topology is parsed by the heterogeneous computing scheduler, and according to the parsing result, the feature extraction operator is assigned to the neural network processor, and according to the parsing result, the classification operator is unloaded to the multi-core central processing unit:

[0084] The heterogeneous computing scheduler calls the graph parsing interface to parse the topology of the computing graph, and obtains the types of each operator (such as convolution, full connection), data dependency relationship, total computing amount (floating point operation times) and input data amount (feature tensor byte number);

[0085] Calculate the computing intensity of each operator: calculate according to the formula "operator computing intensity = operator total computing amount ÷ operator input data amount", set the computing intensity threshold (such as 1000 FLOPS / byte); distribute the operator according to the computing intensity: assign the feature extraction operator (such as convolution, attention operator, computation-intensive) with computing intensity higher than the threshold to the neural network processor (NPU), and unload the classification operator (such as full connection, softmax operator, data-intensive) with computing intensity lower than the threshold to the multi-core central processing unit (CPU); monitor the hardware load rate in real time: according to the formula "hardware load rate = (current allocated operator computing amount ÷ hardware maximum bearable computing amount) x 100%", ensure that the load rate of NPU and CPU is maintained in a reasonable range of 30%-70%, avoid overload or resource idling.

[0086] Step 2.3, for the feature extraction operator assigned to the neural network processor, according to the virtual memory paging mechanism, the access frequency of the model parameters associated with the operator is analyzed, and based on the access frequency analysis result, the high-frequency parameter page is dynamically cached to the on-chip static memory:

[0087] Based on the virtual memory paging mechanism, the model parameters associated with the NPU are divided into a plurality of parameter pages with a fixed size (such as 4KB / page), and the memory controller records the access times of each parameter page in a unit time (such as 1 second);

[0088] Calculate the parameter page access frequency: calculate according to the formula "parameter page access frequency = total access times in unit time ÷ statistical time", set the frequency threshold (such as 5 times / second);

[0089] Dynamically cache high-frequency parameter pages: mark the parameter pages with access frequency higher than the threshold as "hot pages", and migrate them to the on-chip static memory (SRAM) cache through the LRU (Least Recently Used) page replacement algorithm, and the low-frequency parameter pages are retained in the DRAM;

[0090] Evaluate the cache effect: monitor according to the formula "memory cache hit rate = (SRAM hit access times ÷ total access times) x 100%", ensure that the hit rate ≥ 80% to reduce memory access delay.

[0091] Step 2.4, when the neural network processor completes the feature extraction operator execution, according to the cached data page state, the generated feature tensor is transmitted to the memory space of the central processing unit through the direct memory access controller by physical address mapping:

[0092] After the NPU completes feature extraction, the feature tensor is output, and a system is notified through an interrupt signal. The system checks the data page state in the SRAM (such as whether there is an update to be synchronized) and triggers the transmission process after confirming that no additional processing is required.

[0093] Configure the direct memory access controller (DMA): specify the source address of the feature tensor (NPU memory physical address), the target address (CPU DRAM pre-allocated region physical address), and the total data amount (bytes);

[0094] Perform zero-copy transmission: the DMA controller directly transmits data through physical address mapping without CPU mediation. The time consumption is evaluated according to the formula "DMA transmission time = total data amount of the feature tensor ÷ actual transmission bandwidth of the DMA", and the transmission time is ensured to account for ≤20% of the total inference time delay.

[0095] Step 2.5, after the central processing unit receives the transmitted feature tensor, the offloaded classification operator is executed to generate an inference result tensor, and the inference result tensor is output through the edge device:

[0096] The CPU completes interrupt-aware transmission through DMA, and reads the feature tensor data in the DRAM;

[0097] Call the offloaded classification operator (such as the fully connected layer and the softmax layer) to execute calculation with the feature tensor as input, and generate an inference result tensor containing inference results (such as device fault types and confidence levels);

[0098] Through the output interface of the edge device (such as an Ethernet port or a local bus), the inference result tensor is sent to the target terminal (such as a factory monitoring system or an alarm device), and the end-to-end inference output is completed.

[0099] In a preferred embodiment of the present application, the above step 3 can include:

[0100] Step 3.1, after receiving the inference result tensor output by the edge device, it is detected whether the tensor contains a preset abnormal feature vector;

[0101] Step 3.2, when the abnormal feature vector is detected, the teacher model is called to execute inference on the input data that triggered the abnormality detection, and a corresponding reference probability distribution is generated;

[0102] Step 3.3, based on the reference probability distribution, the KL divergence value of the edge device output probability distribution and the reference probability distribution is calculated;

[0103] Step 3.4, generate gradient update instructions with KL divergence value as the optimization target, and add Laplace noise to the instructions;

[0104] Step 3.5, synchronize the noisy gradient update instructions to the edge device to control the edge device to update the network parameters.

[0105] In the embodiments of the present application, through the abnormal driven knowledge transfer and privacy protection mechanism, the dynamic optimization and safe update of the edge model are realized: by detecting the preset abnormal feature vector in the inference result, the insufficient adaptation of the edge model to special scenes (such as sudden failure of industrial equipment) can be captured in time, and the abnormal situation can be accurately identified; the teacher model is called to generate a benchmark probability distribution, which provides a high confidence optimization reference for the edge model, and ensures the reliability of the update direction; the KL divergence value is used to quantify the difference between the edge output and the benchmark distribution, which provides a clear optimization target for gradient update, and ensures that the model parameter adjustment can effectively reduce the performance gap; Laplace noise is added to the gradient update instructions to protect sensitive data (such as device running details) through differential privacy technology while realizing knowledge transfer, avoiding privacy leakage; finally, the noisy instructions are synchronized to the edge device to complete parameter update, so that the edge model can dynamically adapt to abnormal scenes, improving inference accuracy while ensuring data security.

[0106] In the embodiments of the present application, the specific steps include:

[0107] Step 3.1, preset abnormal feature vector library: based on the historical abnormal data (such as rare failure mode, high-risk feature combination) of the target task (such as industrial equipment fault diagnosis), analyze and extract the abnormal feature vector template (such as vibration feature value beyond the normal range, mechanical arm posture vector at a specific angle), and set the key threshold of the feature vector (such as numerical range, pattern similarity standard);

[0108] Abnormality detection: receive the inference result tensor (including classification result and feature vector) output by the edge device, and use the vector similarity matching algorithm (such as cosine similarity) to compare the feature vector in the tensor with the preset abnormal feature vector library; if the similarity of a feature vector to the template in the library is higher than the set threshold (such as 0.8), it is determined that the tensor contains the preset abnormal feature vector, and the subsequent update process is triggered.

[0109] Step 3.2, construction of teacher model: based on a large amount of labeled data, the teacher model needs to be trained to ensure high accuracy and stability on the target task (such as image classification, anomaly detection, etc.), which serves as a "benchmark" to judge whether the inference result of the edge lightweight model is reliable.

[0110] Pre-set of abnormal feature vector: According to the task requirements, by analyzing the historical abnormal data or defining the key feature threshold (such as the feature value exceeding the normal range, the specific mode of feature combination, etc.), the feature vector template for triggering abnormal detection is pre-set.

[0111] Tracing input data: Locating the original input data (such as device vibration sensor time series data, industrial camera image frames) that triggers abnormal detection, preprocessing the data (such as denoising, standardization, format conversion), ensuring that the input format matches the teacher model;

[0112] Generating baseline probability distribution: The teacher model output contains the probability distribution of each target category (such as "bearing wear: 95%, normal: 5%"), which serves as a benchmark to measure the accuracy of the edge model output.

[0113] Step 3.3, extracting edge probability distribution: From the inference result tensor of the edge device, the output probability distribution is parsed (such as the edge model's prediction of "bearing wear: 60%, circuit overheating: 30%, normal: 10%").

[0114] Calculating KL divergence: According to the formula "KL divergence value = the sum of the probability of each category in the edge probability distribution × (the logarithmic ratio of the probability of that category in the edge distribution to the probability in the teacher model baseline distribution)", that is, for each category i, calculate "P_edge(i) × ln(P_edge(i) / P_teacher(i))", and then sum up to get the total divergence value.

[0115] Divergence significance: The larger the KL divergence value, the more significant the difference between the edge model and the teacher model output, which needs to be reduced through parameter updating.

[0116] Step 3.4, generating gradient update instructions: Taking the minimization of the KL divergence value calculated in step 3.3 as the optimization goal, the gradient (i.e. parameter adjustment direction and amplitude) of each layer parameter of the edge model (such as attention head weight, feedforward network bias) is calculated through the backpropagation algorithm, forming the gradient update instructions.

[0117] Adding Laplace noise: To protect data privacy, noise is generated according to the Laplace distribution, and the probability density function of the noise value is , where is the scale parameter (set according to the privacy budget, such as =0.1), is the noise value, is the natural constant; The generated noise is superimposed into the corresponding parameter adjustment amount of the gradient update instruction, which blurs the original data-related information and avoids privacy leakage.

[0118] Step 3.5, secure transmission: transmit the noise-added gradient update instruction to the edge device through an encrypted communication protocol (such as TLS1.3), accompanied by an integrity check code (such as SHA-256 hash) to prevent tampering of the instruction;

[0119] Parameter update: the edge device receives the instruction and verifies the check code, parses the parameter adjustment information in the instruction after passing, calls the model parameter update interface, and performs incremental update on the network parameters (such as pruned and quantized weights and biases) of the lightweight model according to the adjustment amount;

[0120] Confirmation feedback: after the update is completed, the edge device returns a parameter update confirmation signal, and the system records the update time, KL divergence change amount and other logs for evaluating the update effect.

[0121] In a preferred embodiment of the present application, the above step 4 can include:

[0122] Step 4.1, receive the updated network parameters, synchronize the calculation load matrix and task dependency data reported by the edge node in real time, generate operator execution order topology data, and extract the maximum time delay threshold of the critical path combined with the time delay constraint;

[0123] Step 4.2, convert the critical path time delay threshold into resource dimension constraint conditions, construct a three-dimensional resource space coordinate system, the computing power axis is normalized according to the actual available floating point computing power proportion of the neural network processor and the central processing unit; the memory axis is mapped to the continuous virtual address space based on the hierarchical storage architecture by mapping the on-chip static memory and dynamic random memory capacity; the network input and output axis is quantified based on the product of the transmission bandwidth and the reciprocal of the end-to-end time delay;

[0124] Step 4.3, according to the updated network parameters and the calculation load matrix, calculate the resource demand vector and project it to the established three-dimensional space, the computing power demand vector is determined by the peak calculation amount of the model operator; the memory demand vector is determined by the sum of the model parameter volume and the peak occupation of the activation tensor; the network input and output demand vector is determined by the transmission data amount of the feature tensor divided by the critical path time delay threshold, and a multi-dimensional hypercube with time delay constraint is constructed in the three-dimensional space, completing the mapping of resources to the constraint space.

[0125] In the embodiment of the application, by constructing a multi-dimensional resource mapping mechanism with time delay constraints, accurate matching of edge node resource requirements and hardware capabilities is achieved, significantly improving the scientificity and adaptability of resource scheduling in complex scenarios: based on the updated parameters, real-time load matrix and task dependency, operator topology data are generated, combined with time delay constraints to extract the maximum time delay threshold of the critical path, ensuring that the constraint conditions fit the actual running scenario and avoiding blind resource allocation that is divorced from the load; the time delay threshold is converted into a constraint condition for a three-dimensional resource space (computing power, memory, network IO), and the resource dimensions are unified through normalization and quantization benchmarks, solving the problem of difficult direct comparison of heterogeneous resources (NPU / CPU computing power, hierarchical storage, transmission bandwidth), so that resource characteristics can be intuitively measured; the resource requirement vector is calculated and projected into a three-dimensional space, a hypercube with time delay constraints is constructed, and the correlation between the specific requirements of the model for computing power, memory, and network and the time delay limit is clearly presented, providing a spatial model for subsequent resource optimization and matching. The overall process converts abstract resource requirements and time delay constraints into quantifiable and mappable spatial coordinates, realizing the transition from "experience scheduling" to "data-driven scheduling", and adapting to dynamic load and heterogeneous hardware characteristics of edge nodes.

[0126] In the embodiment of the application, the specific steps include:

[0127] Step 4.1, receive the updated network parameters (reflecting the current computing characteristics of the model), and at the same time obtain the computing load matrix (including NPU / CPU available floating-point computing power, load rate, and other hardware state data) and task dependency data (such as a dependency relationship list of "operator A output→operator B input") reported by the edge node in real time;

[0128] Based on the task dependency data, for the operator dependency relationship (in a directed acyclic graph, the edge represents the dependency of "preceding operator→subsequent operator", that is, the output of the preceding operator is the input of the subsequent operator): count the "in-degree" of each operator (node): the number of other operators that the operator depends on (that is, the total number of edges pointing to the operator, for example, if operator B depends on operators A and C, then the in-degree of B is 2).

[0129] Construct an "adjacency list": record the "subsequent operator" of each operator (that is, the operator that the output of the operator points to, for example, if the output of A is the input of B and D, then the adjacency list of A contains B and D).

[0130] Find all operators with in-degree 0 (i.e. operators that do not depend on any other operators and can be executed as a starting point), and add these operators to a queue. Initialize an empty "topology sequence" (for storing the execution order of operators), while the queue is not empty, take an operator from the queue (denoted as the current operator), add it to the topology sequence (indicating that the operator can be executed), and traverse all the subsequent operators of the current operator in the adjacency list, for each subsequent operator: reduce its in-degree by 1 (because the current operator has been executed, and one of the dependent conditions of the subsequent operator has been satisfied), if the in-degree of the subsequent operator is reduced to 0 (i.e. all its dependent operators have been executed), add it to the queue.

[0131] In combination with the preset end-to-end total latency constraint (such as 50ms in an industrial scenario), calculate the maximum latency allowed by the critical path (such as 50ms x 0.6 = 30ms) according to the formula "critical path maximum latency threshold = end-to-end total latency constraint x critical path proportion coefficient (preset, such as 0.6)"; identify the longest execution path from input to output in the topology graph (critical path), and set its maximum allowed latency as the threshold.

[0132] Step 4.2, convert the critical path maximum latency threshold (such as 30ms) into a constraint condition in the resource dimension (such as resource allocation needs to satisfy critical path latency ≤ 30ms);

[0133] Construct a three-dimensional resource space coordinate system:

[0134] Compute the actual available floating-point computing power of the edge node NPU and CPU (such as NPU available 100GFLOPS, CPU available 20GFLOPS, total available 120GFLOPS), and calculate the normalized value of a certain hardware according to the formula "normalized value of a certain hardware = available computing power of the hardware ÷ total available computing power" (NPU: 100 ÷ 120 ≈ 0.83, CPU: 20 ÷ 120 ≈ 0.17), and map the axis range to 0-1;

[0135] Compute the hierarchical storage capacity (such as SRAM 1MB + DRAM 100MB, total 101MB), and calculate the mapping value of a certain storage unit according to the formula "mapping value of a certain storage unit = capacity of the unit ÷ total storage capacity" (SRAM: 1 ÷ 101 ≈ 0.01, DRAM: 100 ÷ 101 ≈ 0.99), and map the axis range to a continuous virtual address space of 0-1;

[0136] Compute the quantization value according to the formula "quantization value = transmission bandwidth (bytes / s) x inverse of end-to-end latency (1 / s)" (such as bandwidth 100MB / s, latency 30ms = 0.03s, quantization value = 100 x (1 ÷ 0.03) ≈ 3330MB / (s 2 ), and then normalize it to the range of 0-1 as a quantitative benchmark of network transmission capability.

[0137] Step 4.3, computing power requirement vector: based on the updated network parameters, the peak computing capacity of all operators (such as convolution operator 50 GFLOPS, fully connected operator 5 GFLOPS) is calculated, and the value of the computing power requirement vector is obtained by the formula "computing power requirement vector value = the sum of the peak computing capacity of each operator"; memory requirement vector: calculated by the formula "memory requirement vector value = model parameter volume + peak activation tensor occupation" (such as parameter 20 MB + peak activation tensor 30 MB = 50 MB); network IO requirement vector: the total transmission data amount of feature tensor on the critical path (such as 10 MB) is calculated, and the value of the network IO requirement vector is calculated by the formula "network IO requirement vector value = transmission data amount ÷ critical path maximum latency threshold" (10 MB ÷ 0.03 s ≈ 333.3 MB / s);

[0138] Space projection and hypercube construction: project the above three requirement vector values to the three-dimensional resource space according to the normalization rules of step 4.2 to obtain the demand point coordinates; construct a multi-dimensional hypercube with latency constraints in the three-dimensional space with the point as the center and in combination with the upper and lower limits of resources corresponding to the critical path latency threshold (such as computing power requirement ≤ 80% of total available computing power), complete the mapping from actual resource requirement to constraint space.

[0139] In a preferred embodiment of the present application, the above step 4 can include:

[0140] Step 4.4, taking the multi-dimensional constraint space as input, recursively performing space bisection along the direction of the normal vector of the latency constraint hyperplane, calculating the variance of each resource dimension, and selecting the dimension with the largest variance as the partition axis; taking the median of the resource vector in the current partition as the cutting point;

[0141] Step 4.5, performing latency compliance verification on the leaf partitions generated by recursive partitioning, marking the partitions that meet the latency threshold as the feasible subspace set, and constructing a local sensitive hash index table in the feasible subspace set, taking the linear combination of resource utilization and latency inverse as the hash key; and performing vector bucketing according to the Pareto frontier distance;

[0142] Step 4.6, inputting the real-time resource requirement vector into the hash index table, performing approximate nearest neighbor search, and minimizing the weighted sum of resource idle rate and latency margin; extracting the non-dominated solution set from the same hash bucket, performing Pareto level sorting on the candidate solution set, and selecting the highest level solution as the final resource allocation scheme.

[0143] In the embodiments of the present application, through efficient space division, accurate feasible subspace screening and intelligent final solution search: the multi-dimensional space is recursively divided along the normal vector of the delay constraint hyperplane, the dimension with the maximum variance is selected as the division axis and the median is cut, the dimension with the most significant resource distribution difference can be focused on, the space is finely divided, and the invalid search range is reduced; the leaf partition is verified for delay compliance, only the feasible subspace that meets the threshold is retained, the subsequent search is focused on the effective solution, and the resource utilization rate and the delay characteristic are associated by local sensitive hash index and Pareto frontier distance bucketing, which greatly shortens the search time consumption; the real-time demand is quickly matched through approximate nearest neighbor search, the weighted sum of resource idle rate and delay margin is minimized, and the Pareto level sorting of non-dominated solution set is combined. The overall process takes into account the accuracy of space division, can quickly adapt to the dynamic load of edge nodes and heterogeneous resource characteristics, and provides a matching scheme that takes into account low latency and high resource utilization for real-time services (such as industrial equipment fault diagnosis).

[0144] In the embodiments of the present application, the specific steps include:

[0145] Step 4.4, taking the multi-dimensional constraint space as input, recursively performing space binary division along the direction of the normal vector of the delay constraint hyperplane: calculating the variance of each resource dimension (such as computing power, memory, network IO), the calculation method is "the variance of a certain resource dimension = the square sum of the difference between all resource vector values in this dimension and the average value of this dimension, divided by the number of vectors", and the dimension with the maximum variance is selected as the division axis; when determining the cutting point, the median of the resource dimension corresponding to the division axis in the current partition is calculated, the calculation method is "sorting all resource vector values in the current partition in this dimension from small to large, and taking the value at the middle position (if the number of vectors is even, taking the average value of the middle two values)", and the space is divided with this median as the cutting point.

[0146] Step 4.5, verifying the leaf partition generated by recursive division for delay compliance, and marking the partition that meets the delay threshold as the feasible subspace set; constructing a local sensitive hash index table in the feasible subspace set: the hash key is a linear combination of resource utilization rate and delay reciprocal, the resource utilization rate is calculated as "the amount of allocated resources in a certain dimension ÷ the total available resources in that dimension", the delay reciprocal is calculated as "1 ÷ the actual execution delay (seconds) of the resource matching scheme", and the hash key is specifically "(resource utilization rate × weight 1) + (delay reciprocal × weight 2)" (weight 1 and weight 2 are preset priority coefficients, which are set according to scene requirements, and the sum is usually 1); the vectors are binned according to the Pareto frontier distance, which is calculated as "the square root of the square sum of the difference between the resource vector and the nearest point on the Pareto frontier in each resource dimension".

[0147] Step 4.6, input the real-time resource demand vector into the hash index table, perform an approximate nearest neighbor search, and minimize the weighted sum of resource idle rate and latency margin in the search process: wherein the resource idle rate is calculated as "1-resource utilization rate", the latency margin is calculated as "actual latency of the resource allocation scheme-maximum latency threshold of the critical path (if the result is negative, take 0)", and the objective function value is "(resource idle rate*weight 3)+(latency margin*weight 4)" (weight 3 and 4 are dynamically adjusted according to the resource tension degree, and the sum is usually 1); extract the non-dominated solution set from the same hash bucket, perform Pareto ranking on the candidate solution set, and select the highest ranked solution as the final resource allocation scheme.

[0148] In a preferred embodiment of the present application, the above step 5 can include:

[0149] Step 5.1, receive the Pareto final resource allocation scheme and parse the computing power allocation ratio, memory space division and network input / output bandwidth quota in the scheme to generate parsed resource allocation data;

[0150] Step 5.2, based on the resource allocation data, calculate the runtime constraints of the instance, allocate the neural network processor computing power quota and on-chip static memory page frame number for the feature extraction operator instance, allocate the central processing unit core binding strategy and dynamic random access memory upper limit for the classification operator instance, and set the inter-container communication bandwidth threshold based on the network input / output axis quantization value;

[0151] Step 5.3, based on the set runtime constraints, dynamically monitor the end-to-end inference latency of the edge device; when the real-time latency deviates from the service level agreement threshold, trigger the elastic scaling operation of the container instance;

[0152] Step 5.4, based on the elastic scaling operation, update the network parameters in real time through the container runtime interface to ensure the consistency of the model state during the elastic scaling process, so as to maintain the end-to-end inference latency within the service level agreement threshold range.

[0153] In the embodiment of the present application, by performing the landing execution and dynamic optimization of the Pareto final resource allocation scheme, the accurate scheduling of edge device resources and the stable control of inference time delay are realized: the computing power, memory and network quota in the final scheme are analyzed to provide clear basis for resource allocation and ensure that the allocation direction is consistent with the final solution; based on the analysis data, the hardware resources (NPU computing power, CPU core binding, storage page frame, etc.) and communication bandwidth threshold are configured for feature extraction operators and classification operators respectively to realize the accurate matching of operators and hardware characteristics and improve the resource utilization; the end-to-end inference time delay is dynamically monitored, and when it deviates from the service level agreement (SLA) threshold, the elastic scaling is triggered to quickly respond to load fluctuations (such as sudden data volume growth) and avoid time delay exceeding the standard; the container runtime interface is used to inject updated parameters during the scaling process to ensure the consistency of model state and prevent fluctuations in inference accuracy or time delay during the scaling process. The overall process forms a closed loop of "final scheme landing-dynamic monitoring-elastic adjustment-state stability", which can not only fully utilize the efficiency of the final resource allocation, but also adapt to environmental changes through dynamic adaptation to ensure that the end-to-end inference time delay is continuously and stably within the SLA threshold.

[0154] In the embodiment of the present application, the specific steps include:

[0155] Step 5.1, receiving the output Pareto optimal resource allocation scheme, extracting the core allocation parameters through the scheme analysis interface:

[0156] Computing power allocation ratio: such as NPU computing power ratio 70%, CPU computing power ratio 30% (based on total available floating point computing power of edge node); memory space division: such as allocating 20% capacity of on-chip static memory (SRAM) to feature extraction operators and 60% capacity of dynamic random access memory (DRAM) to classification operators; network input and output bandwidth quota: such as the communication bandwidth quota between containers (NPU container and CPU container) is 50% of the total available bandwidth;

[0157] The above parameters are converted into structured data (such as JSON format) to generate parsed resource allocation data as the basis for subsequent resource calculation and allocation.

[0158] Step 5.2, according to the formula "NPU computing power quota = total available NPU computing power of edge node x NPU computing power allocation ratio", such as total available NPU computing power 100 GFLOPS, allocation ratio 70%, then quota = 100 x 70% = 70 GFLOPS; according to the formula "SRAM page frame number = total SRAM page frame number x SRAM space allocation ratio", such as total SRAM page frame 1024 pages, allocation ratio 20%, then page frame number = 1024 x 20% ≈ 205 pages, used for caching high-frequency parameters;

[0159] Classification operator instance resource configuration: allocate core binding rules according to CPU core allocation strategy (such as binding physical cores 1-3) to avoid core switching overhead; set memory usage threshold (exceeding triggers OOM protection) according to DRAM allocation proportion (such as 60%);

[0160] According to the formula "inter-container communication bandwidth threshold = total available network bandwidth of edge node x network input / output bandwidth quota proportion", if the total available bandwidth is 100 MB / s and the quota proportion is 50%, the threshold is 100 x 50% = 50 MB / s, ensuring that the transmission does not exceed the limit.

[0161] Step 5.3, real-time latency calculation: deploy a latency monitoring module, and accumulate key node time consumption according to the formula "real-time end-to-end latency = feature extraction time consumption + data transmission time consumption + classification inference time consumption" (such as 20 ms for feature extraction + 10 ms for transmission + 15 ms for classification = 45 ms);

[0162] SLA threshold deviation judgment: according to the formula "upper limit threshold of latency = SLA threshold x (1 + preset deviation proportion, such as 10%)", if the SLA threshold is 50 ms, the upper limit is 50 x 1.1 = 55 ms; according to the formula "lower limit threshold of latency = SLA threshold x (1 - preset deviation proportion, such as 30%)", if the lower limit is 50 x 0.7 = 35 ms;

[0163] Trigger scaling: if the real-time latency exceeds the upper limit for 3 consecutive times (such as > 55 ms), increase NPU / CPU instances (such as NPU instances from 1 to 2); if it is lower than the lower limit (such as < 35 ms), reduce redundant instances (such as NPU instances from 2 to 1).

[0164] Step 5.4, parameter synchronization: load the optimized network parameters in step 3.5 to the newly started container instance through the container runtime interface (such as CRI-O), and replace the old parameters through the hot update interface for the inventory instance;

[0165] State consistency verification: according to the formula "model state consistency error rate = |inference result after scaling - inference result before scaling| ÷ inference result before scaling x 100%", if the result before scaling is 90% and the result after scaling is 89.5%, the error rate is |89.5% - 90%| ÷ 90% x 100% ≈ 0.56%, ensuring that the error is < 1%;

[0166] Closed-loop optimization: continuously monitor the end-to-end latency after scaling, and if it still deviates from the SLA threshold (such as > 55 ms or < 35 ms), repeat steps 5.3-5.4 to adjust until the latency is stable within the range of 45-55 ms.

[0167] For example, Figure 2As shown, the embodiments of the present application also provide an AI large model lightweight deployment system for complex scenarios, comprising:

[0168] The acquisition module is configured to perform structured pruning on the pre-trained Transformer network based on the attention head importance score, dynamically sparsify the feedforward network activation state according to the input tensor entropy value, and adopt dynamic mixed precision quantization to obtain the pruned and quantized network parameters.

[0169] The inference module is configured to deploy the pruned and quantized network parameters to an edge computing device, distribute feature extraction operators to a neural network processor through a heterogeneous computing scheduler, and offload classification operators to a multi-core central processing unit; manage on-chip memory in combination with a virtual memory paging mechanism, implement zero-copy data transmission using a direct memory access controller, and output an inference result tensor.

[0170] The computing module is configured to, when detecting that the inference result tensor contains a preset abnormal feature vector, migrate knowledge by minimizing the difference between the output probability distribution of the edge device and the output probability distribution of the teacher model, perform differential privacy gradient synchronous update, and obtain updated network parameters; map edge node computing power, memory, and network input and output resources to a multi-dimensional constraint space based on the updated network parameters, real-time computing load matrix, task dependency data, and delay constraint; recursively construct a multi-dimensional tree index to locate a partition that meets the delay, and adopt approximate nearest neighbor search to match a Pareto final resource allocation.

[0171] The processing module is configured to dynamically allocate computing instance resources based on the final resource allocation and maintain end-to-end latency within a service level agreement threshold.

[0172] The above describes the preferred embodiments of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered within the scope of protection of the present application.

Claims

1. An AI large model lightweight deployment method for complex scenes, characterized in that, The method comprises: Step 1, performing structured pruning on a pre-trained Transformer network based on attention head importance score, dynamically sparsifying feedforward network activation state according to input tensor entropy value, adopting dynamic mixed precision quantization to obtain pruned and quantized network parameters; Step 2, deploying the pruned and quantized network parameters to an edge computing device, distributing feature extraction operators to a neural network processor through a heterogeneous computing scheduler, and offloading classification operators to a multi-core central processing unit; combining a virtual memory paging mechanism to manage on-chip memory, using a direct memory access controller to realize zero-copy data transmission, and outputting an inference result tensor; Step 3, when detecting that the inference result tensor contains a preset abnormal feature vector, migrating knowledge by minimizing the difference between the output probability distribution of the edge device and the output probability distribution of the teacher model, performing differential privacy gradient synchronization update to obtain updated network parameters; Step 4, based on the updated network parameters, real-time computing load matrix, task dependency data and delay constraint, mapping edge node computing power, memory and network input and output resources into a multi-dimensional constraint space; recursively constructing a multi-dimensional tree index to locate a partition that meets the delay, and using approximate nearest neighbor search to match the Pareto final resource allocation; Step 5, based on the final resource allocation, dynamically allocating computing instance resources to maintain end-to-end delay within the threshold of a service level agreement.

2. The AI large model lightweight deployment method for complex scenes according to claim 1, characterized in that, Performing structured pruning on a pre-trained Transformer network based on attention head importance score, dynamically sparsifying feedforward network activation state according to input tensor entropy value, and adopting dynamic mixed precision quantization to obtain pruned and quantized network parameters, comprising: Step 1.1, performing structured pruning on a pre-trained Transformer network based on attention head importance score to generate a pruned network model; Step 1.2, inputting the pruned network model into a feedforward network, dynamically sparsifying activation state according to the entropy value of the input tensor, and generating a feedforward subnetwork with sparse activation mode; Step 1.3, performing dynamic mixed precision quantization on the weight tensor of the feedforward subnetwork with sparse activation mode, adaptively allocating bit width based on layer sensitivity analysis, and generating three-stage compressed lightweight network parameters.

3. The AI large model lightweight deployment method for complex scenes according to claim 2, characterized in that, Deploying the pruned and quantized network parameters to an edge computing device, distributing feature extraction operators to a neural network processor through a heterogeneous computing scheduler, and offloading classification operators to a multi-core central processing unit; Combining a virtual memory paging mechanism to manage on-chip memory, using a direct memory access controller to realize zero-copy data transmission, and outputting an inference result tensor, comprising: Step 2.1, in response to a deployment instruction, loading the pruned and quantized network parameters to a heterogeneous memory architecture of an edge computing device; Step 2.2, based on the loaded network parameters, analyzing the model computation graph topology through a heterogeneous computing scheduler, and distributing feature extraction operators to a neural network processor according to the analysis result, while offloading classification operators to a multi-core central processing unit according to the analysis result; Step 2.3, according to the virtual memory paging mechanism, the access frequency of the model parameters associated with the feature extraction operator allocated to the neural network processor is analyzed, and based on the analysis result, the high-frequency parameter pages are dynamically cached to the on-chip static memory; Step 2.4, after the neural network processor completes the execution of the feature extraction operator, according to the cached data page state, the generated feature tensor is transmitted to the memory space of the central processing unit through the direct memory access controller via physical address mapping; Step 2.5, after the central processing unit receives the transmitted feature tensor, the unloaded classification operator is executed to generate an inference result tensor, and the inference result tensor is output through the edge device.

4. The AI large model lightweight deployment method for complex scenes according to claim 3, characterized in that, When it is detected that the inference result tensor contains a preset abnormal feature vector, the knowledge is migrated by minimizing the difference between the edge device output and the teacher model output probability distribution, the differential privacy gradient is updated synchronously, and the updated network parameters are obtained, including: Step 3.1, after receiving the inference result tensor output by the edge device, it is detected whether the tensor contains a preset abnormal feature vector; Step 3.2, when the abnormal feature vector is detected, the teacher model is called to execute inference for the input data triggering the abnormal detection, and a corresponding benchmark probability distribution is generated; Step 3.3, based on the benchmark probability distribution, the KL divergence value of the edge device output probability distribution and the benchmark probability distribution is calculated; Step 3.4, taking the KL divergence value as the optimization target, generating gradient update instructions, and adding Laplace noise to the instructions; Step 3.5, synchronizing the noisy gradient update instructions to the edge device to control the edge device to update the network parameters according to the instructions.

5. The AI large model lightweight deployment method for complex scenes according to claim 4, characterized in that, Based on the updated network parameters, real-time computing load matrix, task dependency data and delay constraint, the edge node computing power, memory and network input and output resources are mapped into a multi-dimensional constraint space, including: Step 4.1, receiving the updated network parameters, synchronously analyzing the computing load matrix and task dependency data reported by the edge node in real time, generating an execution order topology data between operators, and extracting a key path maximum delay threshold value combined with the delay constraint; Step 4.2, convert the key path delay threshold value into a resource dimension constraint condition, construct a three-dimensional resource space coordinate system, the computing power axis is normalized according to the actual available floating point computing power ratio of the neural network processor and the central processing unit; the memory axis is mapped to a continuous virtual address space based on the hierarchical storage architecture; the network input and output axis takes the product of the transmission bandwidth and the reciprocal of the end-to-end delay as the quantitative benchmark; Step 4.3, according to the updated network parameters and the computing load matrix, the resource demand vector is calculated and projected into the established three-dimensional space, the computing power demand vector is determined by the peak calculation amount of the model operator; the memory demand vector is determined by the sum of the model parameter volume and the peak occupation of the activation tensor; the network input and output demand vector is determined by the data amount of the feature tensor transmission divided by the key path delay threshold value, and a multi-dimensional hypercube with delay constraint is constructed in the three-dimensional space, completing the mapping of resources to the constraint space.

6. The AI large model lightweight deployment method for complex scenes according to claim 5, characterized in that, Recursive construction of multi-dimensional tree index positioning meets partition of delay, adopts approximate nearest neighbor search to match the final resource allocation of Pareto, including: Step 4.4, taking the multi-dimensional constraint space as input, recursively performing space bisection along the normal vector direction of the delay constraint hyperplane, calculating the variance of each resource dimension, and selecting the dimension with the largest variance as the splitting axis; Taking the median of the resource vector in the current partition as the cutting point; Step 4.5, the leaf partition generated by recursive partition is verified for delay compliance, and the partition that meets the delay threshold is marked as a feasible subspace set. In the feasible subspace set, a local sensitive hash index table is constructed, and the linear combination of resource utilization rate and delay inverse is used as the hash key; According to the distance of the Pareto frontier, the vector is divided into buckets; Step 4.6, input the real-time resource demand vector into the hash index table, perform approximate nearest neighbor search, and minimize the weighted sum of resource idle rate and delay margin; Extract the non-dominated solution set from the same hash bucket, and sort the candidate solution set according to the Pareto level, and select the highest level solution as the final resource allocation scheme.

7. The AI large model lightweight deployment method for complex scenes according to claim 6, characterized in that, Based on the final resource allocation, dynamically allocate instance resources to maintain end-to-end latency within the service level agreement threshold, including: Step 5.1, receive the Pareto final resource allocation scheme and parse the computing power allocation ratio, memory space division and network input / output bandwidth quota in the scheme to generate parsed resource allocation data; Step 5.2, based on the resource allocation data, calculate the runtime constraints of the instance, allocate the neural network processor computing power quota and on-chip static memory page frame number for the feature extraction operator instance; Assign the central processing unit core binding strategy and dynamic random access memory upper limit for the classification operator instance; And set the inter-container communication bandwidth threshold based on the network input / output axis quantization value; Step 5.3, based on the set runtime constraints, dynamically monitor the end-to-end inference latency of the edge device; When the real-time latency deviates from the service level agreement threshold, trigger the elastic scaling operation of the container instance; Step 5.4, based on the elastic scaling operation, real-time injection of updated network parameters through the container runtime interface ensures model state consistency during elastic scaling, thereby maintaining end-to-end inference latency within the service level agreement threshold range.

8. An AI large model lightweight deployment system for complex scenarios, the system implements the method of any one of claims 1 to 7, characterized in that, Including: The acquisition module is used to perform structured pruning on the pre-trained Transformer network based on the attention head importance score, dynamically sparsify the feedforward network activation state according to the input tensor entropy value, and obtain the pruned and quantized network parameters by using dynamic mixed precision quantization; The inference module is used to deploy the pruned and quantized network parameters to the edge computing device, and assign the feature extraction operator to the neural network processor through the heterogeneous computing scheduler, and unload the classification operator to the multi-core central processing unit; Combine the virtual memory paging mechanism to manage on-chip memory, and use the direct memory access controller to realize zero-copy data transmission, and output the inference result tensor; The computing module is configured to, when it is detected that the inference result tensor contains a preset abnormal feature vector, migrate knowledge by minimizing the difference between the edge device output and the teacher model output probability distribution, perform differential privacy gradient synchronization update, and obtain updated network parameters; map edge node computing power, memory, and network input and output resources to a multi-dimensional constraint space based on the updated network parameters, real-time computation load matrix, task dependency data, and delay constraints; A multi-dimensional tree index is recursively constructed to position a partition that meets a delay, and an approximate nearest neighbor search is used to match a Pareto final resource allocation; The processing module is configured to dynamically allocate computing instance resources based on the final resource allocation and maintain an end-to-end delay within a service level agreement threshold.

9. A computing device, comprising: The method comprises the following steps: one or more processors; a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images