Edge device oriented machine learning model hierarchical pruning compression inference acceleration method

Abstract

The application discloses an edge device-oriented machine learning model hierarchical pruning compression inference acceleration method, comprising S1, obtaining the hardware configuration parameters of a target edge device, the hardware configuration parameters comprising the peak computing power of a computing unit, the memory bandwidth, the cache capacity of each level and the bus transmission rate; for a machine learning model to be compressed, computing features are extracted layer by layer according to the model structure; according to the hardware configuration parameters and the computing features, an end-side inference delay prediction model is constructed to predict the inference delay of each layer and the whole model under different pruning configurations; the application has the beneficial effects that by constructing the hardware-aware end-side inference delay prediction model, the application takes delay prediction as the direct feedback of pruning search, and takes the weighted combination of the minimum inference delay and the maximum model accuracy as the multi-objective optimization objective function, thereby realizing the deep fusion and joint optimization of the hardware index and the accuracy index, and overcoming the defects of the prior art that the two are processed separately.

Description

Technical Field

[0001] This invention relates to the field of edge computing and deep learning model compression technology, specifically a hierarchical pruning compression inference acceleration method for machine learning models for edge devices. Background Technology

[0002] With the rapid development of deep learning technology, machine learning models have achieved remarkable results in fields such as computer vision, natural language processing, and speech recognition. However, these models typically contain millions or even billions of parameters, resulting in high computational and storage costs, which severely restricts their real-time inference deployment on resource-constrained edge devices. To address this issue, model compression technology has emerged, among which pruning serves as an effective method for lightweighting models. By removing redundant channels, filters, or attention heads in the network, it reduces the number of model parameters and computational load, thereby lowering inference latency.

[0003] Currently, pruning methods for edge devices can be mainly categorized as follows: (1) Rule-based or heuristic unified pruning methods. These methods apply the same pruning rate to all layers of the model (e.g., uniformly pruning 50% of the number of channels in each layer), or allocate pruning filters according to simple rules (e.g., the size of the layer parameters). They are simple to implement, but do not consider the different impacts of different network layers on the model's output accuracy. This often leads to over-pruning of sensitive layers, resulting in a severe drop in accuracy, or insufficient pruning of non-sensitive layers, leaving redundancy.

[0004] (2) Layer-by-layer differentiated pruning method based on sensitivity analysis. This type of method assesses the impact of pruning at each layer on model accuracy and assigns different pruning rates to layers with different sensitivities, pruning less for layers with high sensitivity and more for layers with low sensitivity. This method is an improvement over uniform pruning, but its sensitivity assessment is usually based only on a single static analysis and does not establish a correlation with the actual hardware latency of the target edge device, so the acceleration effect of the strategy on the hardware is not guaranteed.

[0005] (4) Hardware-aware model compression methods. In recent years, some studies have begun to incorporate hardware metrics (computing power, memory bandwidth) into the objective function of pruning optimization, and guide pruning decisions by constructing latency prediction models. However, existing hardware-aware methods mostly use static lookup tables or linear regression models to predict inference latency, ignoring the actual impact of inter-layer data dependencies on pipeline parallelism, resulting in limited prediction accuracy. In addition, after generating pruning strategies, these methods lack operator-level adaptation optimization of the compression model on specific hardware, leading to a significant gap between theoretical acceleration and actual inference acceleration.

[0006] In summary, existing technologies either prioritize model accuracy preservation or hardware latency in isolation, failing to construct a multi-objective optimization framework that deeply integrates both. More importantly, existing pruning search strategies lack effective utilization of search history, easily lingering near local optima during iteration and struggling to converge to the globally Pareto-optimal pruning configuration. Existing sensitivity analysis relies solely on changes in model accuracy, neglecting to incorporate the actual latency gains of each pruned layer on the target hardware. This means that parameters retained in highly sensitive layers may not necessarily yield the highest latency gains on the hardware. Furthermore, existing methods typically only accumulate the independently computed operator latency of each layer when predicting the inference latency of the pruned model, failing to consider actual hardware runtime overhead such as pipeline bubbles, cache misses, and bus contention caused by inter-layer data dependencies. This leads to discrepancies between predicted and actual test latency. Significant latency bias affects the accuracy of the pruning search direction. Existing technologies often stop at generating pruned weight files without considering the specific hardware characteristics of the target edge device (such as memory access patterns and parallel granularity of computing units) to perform operator fusion and memory rearrangement on the compressed model. This results in the compressed model not being able to fully utilize the actual computing power of the hardware during deployment. Furthermore, the pruning configuration of the compressed model generated by existing methods is fixed after deployment, making it impossible to dynamically switch inference paths based on the real-time resource status of the edge device during operation. This makes it difficult to continuously guarantee low-latency inference in complex and ever-changing edge environments. To address these issues, we propose a hierarchical pruning and compressed inference acceleration method for machine learning models targeting edge devices. Summary of the Invention

[0007] The purpose of this invention is to provide a hierarchical pruning and compression inference acceleration method for machine learning models for edge devices, in order to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, a hierarchical pruning and compressed inference acceleration method for machine learning models targeting edge devices includes the following steps: S1. Obtain the hardware configuration parameters of the target edge device, including the peak computing power of the computing unit, memory bandwidth, cache capacity at each level, and bus transmission rate; for the machine learning model to be compressed, extract the computational features layer by layer according to the model structure, including the number of floating-point operations, the number of parameters, the amount of memory access, and the data dependencies of each layer; based on the hardware configuration parameters and the computational features, construct an edge-side inference latency prediction model to predict the thrust latency of each layer and the whole model under different pruning configurations; S2. Set the order N of the pruning search, where N is a positive integer greater than or equal to 2; construct a joint search space using the number of channels and weight sparsity of each layer of the model, and perform iterative search in the joint search space using a multi-objective optimization strategy; generate a set of candidate pruned models under the current delay budget in each iteration, and select the Pareto optimal model sequence that satisfies the accuracy constraint from the set of candidate pruned models; during the selection process, the search information of the current iteration also refers to the historical search information of the previous N-1 iterations; S3. Based on the pruning configuration of each layer in the output Pareto optimal model sequence, calculate the sensitivity of each network layer to the overall accuracy of the model; based on the sensitivity, assign a first pruning rate to layers with sensitivity higher than a preset threshold, and assign a second pruning rate to layers with sensitivity lower than a preset threshold, wherein the first pruning rate is less than the second pruning rate; combine the pruning configurations of each layer to generate a differentiated layered pruning strategy. S4. Perform structured pruning on the machine learning model according to the differentiated hierarchical pruning strategy to obtain a compressed model; perform operator fusion and memory layout optimization on the compressed model according to the hardware characteristics of the target edge device to generate a thrust engine adapted to the target edge device; deploy the thrust engine to the target edge device to perform inference tasks. S5. Monitor the real-time resource status of the target edge device during the inference process. The real-time resource status includes CPU utilization, available memory, and device temperature. When the real-time resource status exceeds the preset normal range, switch the currently running compressed model to a pruned version with less computational load according to the Pareto optimal model sequence.

[0009] Preferably, the step of constructing the edge-side inference delay prediction model in step S1 includes: For the hardware architecture of the target edge device, a pre-made measured latency lookup table for multiple basic operator types is prepared. The basic operator types include convolution operator, fully connected operator, pooling operator, and activation operator. Based on the operator type and computational characteristics of each layer of the machine learning model, the baseline latency of each basic operator is obtained by querying the measured latency lookup table. Combined with the data dependency relationship between layers, a pipeline parallel overhead correction term is introduced to calculate the overall inference latency of the model.

[0010] Preferably, the objective function of the multi-objective optimization strategy in step S2 is a weighted combination of minimizing inference latency and maximizing model accuracy, and the constraints include that the number of channels after pruning in each layer is not less than a preset lower limit and the sparsity of the weights in each layer is not higher than a preset upper limit.

[0011] Preferably, the N-order historical search information in step S2 is utilized in the following way: Define the search space for the current iteration as: , , in, For the first The model selected in the next iteration This is a function for evaluating model accuracy. Indicates by the first Sub-iteration model in delayed budget The set of all candidate pruning models generated under constraints.

[0012] Preferably, step S2, which involves selecting the Pareto optimal model sequence from the candidate pruning model set, includes: Calculate the inference latency and accuracy of each candidate model in the candidate pruning model set; use inference latency and accuracy as two-dimensional objectives to select non-dominated solutions to form a Pareto front; select models whose accuracy meets a preset accuracy threshold from the Pareto front as the Pareto optimal model sequence.

[0013] Preferably, step S3, which involves calculating the sensitivity of each network layer to the overall accuracy of the model, includes: For each network layer, calculate the change in accuracy of the model on the validation set before and after pruning the layer; divide the change in accuracy by the pruning ratio of the layer parameters to obtain the normalized sensitivity score of the layer; the sensitivity threshold is dynamically determined based on the statistical distribution of the normalized sensitivity scores of all layers.

[0014] As a preferred embodiment, the differentiated hierarchical pruning strategy in step S3 further includes: According to the order of the network layers in the forward propagation of the model, each layer is divided into multiple stages; a consistent pruning rate constraint is applied to each layer within the same stage; and the differentiated pruning strategy is adopted between different stages.

[0015] Preferably, the operator fusion in step S4 includes fusing the convolutional layer, batch normalization layer, and activation function layer into a single computational unit; the memory layout optimization includes rearranging the weight matrix of the compression model according to the memory access characteristics of the target edge device.

[0016] Preferably, the machine learning model is a convolutional neural network or a Transformer network; the structured pruning is channel pruning, filter pruning, or attention head pruning; and the target edge device is an embedded AI development board, a smart collection device, or an IoT terminal device.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention constructs a hardware-aware edge inference latency prediction model, using latency prediction as direct feedback for pruning search. It employs a weighted combination of minimizing inference latency and maximizing model accuracy as the multi-objective optimization objective function, achieving deep integration and joint optimization of hardware and accuracy metrics, overcoming the shortcomings of existing technologies that treat them separately. Furthermore, it adopts an N-order history memory-guided iterative search strategy (N≥2), where the search space of the current iteration is limited by historical information from the previous N-1 iterations. This effectively utilizes historical experience during the search process, avoiding the problem of traditional greedy search and genetic algorithms easily getting trapped in local optima, thus improving search efficiency and the global optimality of the solution.

[0018] 2. When constructing the latency prediction model, this invention not only obtains the baseline latency of each operator through a lookup table based on actual measurements, but also introduces a pipeline parallel overhead correction term in combination with the data dependency relationship between layers. It takes into account the actual hardware runtime overhead such as pipeline bubbles, cache misses and bus contention, which significantly improves the prediction accuracy of inference latency under pruning configuration.

[0019] 3. Based on the Pareto optimal model sequence, this invention calculates the sensitivity of each layer to the overall accuracy and allocates differentiated pruning rates accordingly (low retention for sensitive layers and high compression for non-sensitive layers). At the same time, it combines the consistency constraint of pruning rate within the stage to maintain the balance of computational load, so that accuracy protection and hardware acceleration form a closed loop linkage, overcoming the problem of the disconnect between existing sensitivity analysis and hardware acceleration.

[0020] 4. After generating the compressed model, this invention further performs operator fusion and memory layout optimization based on the hardware characteristics of the target edge device to generate an inference engine adapted to the hardware, so that the theoretical speedup is effectively transformed into the actual speedup ratio. At the same time, by monitoring the device resource status in real time, when resources are tight, it dynamically switches to a pruned version with less computation based on the Pareto optimal model sequence, giving the model the adaptive ability to continuously and stably perform low-latency inference in complex edge environments. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see Figure 1This invention provides a technical solution: a hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices, comprising the following steps: S1. Obtain the hardware configuration parameters of the target edge device, including the peak computing power of the computing unit, memory bandwidth, cache capacity at each level, and bus transmission rate; for the machine learning model to be compressed, extract the computational features layer by layer according to the model structure, including the number of floating-point operations, the number of parameters, the amount of memory access, and the data dependencies of each layer; based on the hardware configuration parameters and computational features, construct an edge-side inference latency prediction model to predict the thrust latency of each layer and the whole model under different pruning configurations.

[0024] It should be noted that the purpose of this embodiment is to construct a prediction model that can accurately predict the inference latency of the model under different pruning configurations.

[0025] First, obtain the hardware configuration parameters of the target edge device. Taking the Raspberry Pi 4B as an example, its hardware configuration parameters include: a quad-core Cortex-A72 CPU with a peak computing power of approximately 6.4 GFLOPS; a memory bandwidth of approximately 12.8 GB / s; an L2 cache capacity of 1MB; and a bus transfer rate limited by the SD card I / O and USB interface bandwidth. In practical applications, these parameters can be obtained by reading the device's system files or by using hardware detection tools.

[0026] Secondly, for the machine learning model to be compressed, computational features are extracted layer by layer according to the model structure. Taking ResNet-50 as an example, its network structure is traversed layer by layer, and the following computational features are extracted for each layer: number of floating-point operations (FLOPs), number of parameters, memory access, and inter-layer data dependencies. Specifically, for convolutional layers, their FLOPs can be expressed by the formula: Calculation, where and The kernel size is the convolution kernel size. and Number of input / output channels and The output feature map size; the number of parameters can be determined by... Computation; memory access volume includes the total amount of reading and writing of input feature maps, weights, and output feature maps. Inter-layer data dependencies are obtained through the model's computation graph, recording the input source and output destination of each layer.

[0027] Then, based on the hardware configuration parameters and computational characteristics, an edge-side inference latency prediction model is constructed. The construction method of the edge-side inference latency prediction model is as follows: for the hardware architecture of Raspberry Pi 4B, a pre-built measured latency lookup table for various basic operator types is provided. Specifically, a micro-benchmark program is written on Raspberry Pi 4B to conduct actual tests on convolution operators (including different convolution kernel sizes of 3×3, 5×5, 7×7 and different combinations of channel numbers), fully connected operators (different input and output dimensions), pooling operators (different pooling window sizes), and activation operators (ReLU, Sigmoid, Tanh, etc.). The average execution latency of each operator under different input sizes is recorded to form a measured latency lookup table.

[0028] For a given compressed model, the baseline latency of each basic operator is obtained by querying a measured latency lookup table, based on the operator type and computational characteristics of each layer. On this basis, a pipeline parallelism overhead correction term is introduced, incorporating inter-layer data dependencies. Specifically, when the input of a layer depends on the output of the previous layer and data transfer exists between the two layers, a data transfer latency correction term is introduced. ,in This is an empirical coefficient (ranging from 0.3 to 1.0). The amount of data to be transferred. To measure the actual memory copy bandwidth, a parallelism correction term is introduced when two layers can be executed in parallel on different computing units, reducing the time of overlapping execution parts according to parallel efficiency. Finally, the overall model inference latency is calculated by pipeline scheduling of the latency of each layer according to data dependencies, rather than simply accumulating layer by layer.

[0029] Alternatively, the following preferred implementation method can be adopted: For the hardware architecture of the target edge device, pre-set a lookup table of measured latency for multiple basic operator types; Based on the operator type and computational characteristics of each layer of the machine learning model, obtain the baseline latency of each basic operator by querying the measured latency lookup table, and combine the inter-layer data dependency relationship to introduce a pipeline parallel overhead correction term to calculate the overall inference latency of the model.

[0030] Using the above method, the edge-side inference latency prediction model can predict the inference latency of each layer and the whole model under different pruning configurations (different number of channels retained and weight sparsity).

[0031] S2. Set the order N of the pruning search, where N is a positive integer greater than or equal to 2; construct a joint search space using the number of channels and weight sparsity of each layer of the model, and perform iterative search in the joint search space using a multi-objective optimization strategy; generate a set of candidate pruned models under the current delay budget in each iteration, and select the Pareto optimal model sequence that meets the accuracy constraint from the set of candidate pruned models; during the selection process, the search information of the current iteration also refers to the historical search information of the previous N-1 iterations.

[0032] It should be noted that in this embodiment, the aim is to use a multi-objective optimization strategy and N-order historical memory information to search for a Pareto optimal model sequence that satisfies the accuracy constraint.

[0033] The order N of the pruning search is set, where N is a positive integer greater than or equal to 2. In this implementation, N is set to 3, meaning that the current iteration of the search also references the historical search information from the previous two iterations. In practical applications, the value of N can be adjusted according to the model size and hardware resources, generally ranging from 2 to 5. The larger the value of N, the higher the utilization rate of information, but the computational cost also increases accordingly.

[0034] The joint search space is constructed using the number of channels and weight sparsity of each layer of the model. For a model with L layers, the search space is defined as follows: ,in Each dimension is used to store the pruning configurations for each layer, and the search space size is the product of the configuration options for each layer.

[0035] A multi-objective optimization strategy is employed to perform an iterative search in the joint search space. The objective function of the multi-objective optimization strategy is a weighted combination of minimizing inference latency and maximizing model accuracy, specifically expressed as: )+(1- ) ,in For the model Predictive inference latency on the target device, For the original model inference delay, For the model The accuracy of the validation set. The accuracy of the original model. The weighting coefficient (value range 0-1, in this embodiment) (Take 0.5, which can be adjusted according to actual needs). Constraints include: the number of channels after pruning in each layer is not less than the preset lower limit (in this embodiment, it is set to not less than 20% of the original number of channels), and the weight sparsity of each layer is not higher than the preset upper limit (in this embodiment, it is set to not higher than 90%).

[0036] Each iteration generates the current delay budget. The set of candidate pruned models is defined below. Specifically, multiple candidate pruning configurations are generated by randomly sampling in the current search space or based on historical information, with each configuration corresponding to a candidate pruned model. For each candidate model, its inference latency is predicted using the latency prediction model constructed in step S1, and its accuracy is evaluated on the validation set.

[0037] The Pareto optimal model sequence for elm trees that meets the accuracy requirements is selected from the candidate pruning model set. The selection process specifically includes: calculating the thrust delay and accuracy of each candidate model in the candidate pruning model set; using the thrust delay and accuracy as two-dimensional objectives, selecting non-dominated solutions to form the Pareto front; and selecting models from the Pareto front whose accuracy meets a preset accuracy threshold (in this embodiment, it is set to no less than 95% of the accuracy of the original model) as the Pareto optimal model sequence.

[0038] During the filtering process, the search information of the current iteration also references the historical search information of the previous N-1 iterations. In this embodiment, the N-order historical search information is utilized as follows: the search space of the current iteration is defined as: , , in, For the first The model selected in the next iteration This is a function for evaluating model accuracy. Indicates by the first Sub-iteration model in delayed budget The set consists of all candidate pruned models generated under the constraint. This constraint limits the search in the current iteration to the neighborhood of the models from the previous N iterations, while delaying the budget. The adjustment makes the search space more extensible, thus exploring new solution spaces while maintaining the historical optimal direction, effectively avoiding getting trapped in local optima.

[0039] In this implementation, the budget is delayed after each iteration. The system dynamically adjusts according to a predetermined strategy: if no model that meets the accuracy constraints is found in the current iteration, the delay budget is appropriately relaxed; if multiple models that meet the accuracy constraints are found in the current iteration, the delay budget is appropriately tightened in order to gradually approach the optimal delay-accuracy balance point.

[0040] S3. Based on the pruning configuration of each layer in the output Pareto optimal model sequence, calculate the sensitivity of each network layer to the overall accuracy of the model; based on the sensitivity, assign a first pruning rate to layers with sensitivity higher than a preset threshold, and assign a second pruning rate to layers with sensitivity lower than a preset threshold, wherein the first pruning rate is less than the second pruning rate; combine the pruning configurations of each layer to generate a differentiated layered pruning strategy.

[0041] It should be noted that in this embodiment, the aim is to calculate the sensitivity of each layer to the overall accuracy of the model based on the pruning configuration of each layer in the Pareto optimal model sequence, and generate differentiated hierarchical pruning strategies accordingly.

[0042] Based on the pruning configuration of each layer in the Pareto optimal model sequence output in step S2, the sensitivity of each network layer to the overall accuracy of the model is calculated. The specific calculation method is as follows: for each network layer, calculate the change in the accuracy of the model on the validation set before and after pruning the layer; divide the change in accuracy by the pruning ratio of the layer parameters to obtain the normalized sensitivity score of the layer.

[0043] Taking ResNet-50 as an example, for the first convolutional layer, pruning rates of 10%, 20%, 30%, 40%, and 50% are applied to this layer respectively, resulting in multiple pruned model copies. The accuracy of each copy is tested on the validation set, and the change in accuracy relative to the original model is recorded. The change in accuracy is linearly fitted or interpolated with the pruning ratio of this layer to obtain the sensitivity curve of that layer. The above operation is repeated for each layer to obtain the normalized sensitivity score of all layers.

[0044] Sort the normalized sensitivity scores of all layers by numerical value and take the median as the sensitivity threshold.

[0045] Based on sensitivity, a first pruning rate is assigned to layers with sensitivity above a preset threshold, and a second pruning rate is assigned to layers with sensitivity below the preset threshold, with the first pruning rate being less than the second pruning rate. For example, for layers with sensitivity above the threshold (i.e., critical layers that have a significant impact on accuracy), a lower pruning rate (e.g., no more than 20%) is assigned to protect model accuracy; for layers with sensitivity below the threshold (i.e., non-critical layers that have a minor impact on accuracy), a higher pruning rate (e.g., no more than 60%) is assigned to fully compress the model and reduce computational load. The pruning configurations of each layer are combined to generate differentiated layered pruning strategies.

[0046] The differentiated layered pruning strategy also includes: dividing each layer into multiple stages according to the order of the network layers in the forward propagation of the model (e.g., ResNet-50 can be divided into stages according to different residual blocks, with each residual block as a stage); applying a consistent pruning rate constraint to each layer within the same stage to maintain the balance of computational load within the stage and avoid computational bottlenecks caused by excessive pruning of a certain layer within the stage; and adopting the differentiated pruning strategy between different stages.

[0047] S4. Perform structured pruning on the machine learning model according to the differentiated hierarchical pruning strategy to obtain a compressed model; perform operator fusion and memory layout optimization on the compressed model according to the hardware characteristics of the target edge device to generate a thrust engine adapted to the target edge device; deploy the thrust engine to the target edge device to perform inference tasks.

[0048] It should be noted that in this embodiment, the aim is to actually prune the model according to the generated pruning strategy and adapt and optimize it for the target hardware to generate an inference engine that can be deployed efficiently.

[0049] In this embodiment, structured pruning is channel pruning. Specifically, for each convolutional layer, the number of channels to be retained is calculated based on the pruning rate determined in step S3; channels are sorted by importance, and the most important channels are retained while the remaining channels are removed; for the input channels of adjacent layers, adjustments are made based on the output channels of the previous layer to maintain channel correspondence. For Transformer networks, structured pruning can be attention head pruning, i.e., removing redundant attention heads based on their importance scores.

[0050] Operator fusion involves fusing convolutional layers, batch normalization layers, and activation function layers into a single operational unit. Specifically, this is achieved by fusing the parameters of the batch normalization layer (mean, variance, scaling factor, and shift factor) into the weights and biases of the convolutional layer, resulting in... , ,in and For the weights and biases of the convolutional layer, , , , For batch normalized layer parameters, To prevent small constants from being divided by zero, the activation function is merged in-situ with the convolutional layer after fusion, reducing the storage and transport overhead of intermediate tensors.

[0051] For the ARM Neon vector instruction set used by Raspberry Pi 4B, the convolution weight matrix is ​​rearranged according to the NHWC (number of samples-height-width-number of channels) format to improve the continuity of data access and the efficiency of vectorized loading; for devices that support INT8 quantization, the weight matrix is ​​quantized to INT8 format and rearranged into a layout more suitable for SIMD instruction access.

[0052] S5. Monitor the real-time resource status of the target edge device during the inference process. The real-time resource status includes CPU utilization, available memory, and device temperature. When the real-time resource status exceeds the preset normal range, switch the currently running compressed model to a pruned version with less computation based on the Pareto optimal model sequence.

[0053] It should be noted that the purpose of this embodiment is to monitor the real-time resource status of the edge device during the inference process and dynamically switch the inference model according to the resource status.

[0054] Monitoring is achieved by running a background monitoring thread on a Raspberry Pi 4B: CPU usage, available memory, and CPU temperature are read periodically using system commands.

[0055] The default normal range is: CPU utilization < 80%, available memory > 200MB, and device temperature < 80℃. When any of these indicators exceeds the above range, a model switching mechanism is triggered. Specifically, from the Pareto optimal model sequence saved in step S2, a pruned version with lower inference latency and sufficient current resource status is selected for switching. The switching process is achieved by dynamically loading model weights. Multiple pruned versions in the Pareto optimal model sequence have been pre-optimized for hardware adaptation in step S4 and stored in binary format on the edge device's storage medium. During switching, there is no need to recompile the inference engine; only the weight file needs to be replaced and the inference engine context reset. The switching time is controlled within milliseconds.

[0056] If the minimally pruned version is already running and the resource status is still outside the normal range, an alarm will be triggered and the inference frequency will be reduced to ensure the safe operation of the device under extreme conditions.

[0057] In summary, this invention provides accurate feedback for search through latency prediction, ensures strategy quality through vertical memory search, achieves an optimal balance between accuracy and acceleration through differentiated pruning, transforms compression benefits into actual acceleration through hardware adaptation, and ensures stability after deployment through dynamic adjustment. It solves the entire technical challenge of accelerating machine learning model inference on edge devices.

[0058] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A hierarchical pruning and compressed inference acceleration method for machine learning models targeting edge devices, characterized in that, Includes the following steps: S1. Obtain the hardware configuration parameters of the target edge device, including the peak computing power of the computing unit, memory bandwidth, cache capacity at each level, and bus transmission rate; for the machine learning model to be compressed, extract the computational features layer by layer according to the model structure, including the number of floating-point operations, the number of parameters, the amount of memory access, and the data dependencies of each layer; based on the hardware configuration parameters and the computational features, construct an edge-side inference latency prediction model to predict the thrust latency of each layer and the whole model under different pruning configurations; S2. Set the order N of the pruning search, where N is a positive integer greater than or equal to 2; construct a joint search space using the number of channels and weight sparsity of each layer of the model, and perform iterative search in the joint search space using a multi-objective optimization strategy; generate a set of candidate pruned models under the current delay budget in each iteration, and select the Pareto optimal model sequence that satisfies the accuracy constraint from the set of candidate pruned models; during the selection process, the search information of the current iteration also refers to the historical search information of the previous N-1 iterations; S3. Based on the pruning configuration of each layer in the output Pareto optimal model sequence, calculate the sensitivity of each network layer to the overall accuracy of the model; based on the sensitivity, assign a first pruning rate to layers with sensitivity higher than a preset threshold, and assign a second pruning rate to layers with sensitivity lower than a preset threshold, wherein the first pruning rate is less than the second pruning rate; combine the pruning configurations of each layer to generate a differentiated layered pruning strategy. S4. Perform structured pruning on the machine learning model according to the differentiated hierarchical pruning strategy to obtain a compressed model; perform operator fusion and memory layout optimization on the compressed model according to the hardware characteristics of the target edge device to generate a thrust engine adapted to the target edge device; deploy the thrust engine to the target edge device to perform inference tasks. S5. Monitor the real-time resource status of the target edge device during the inference process. The real-time resource status includes CPU utilization, available memory, and device temperature. When the real-time resource status exceeds the preset normal range, switch the currently running compressed model to a pruned version with less computational load according to the Pareto optimal model sequence.

2. The hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices according to claim 1, characterized in that, The step of constructing the end-side inference delay prediction model in step S1 includes: For the hardware architecture of the target edge device, a pre-made measured latency lookup table for multiple basic operator types is prepared. The basic operator types include convolution operator, fully connected operator, pooling operator, and activation operator. Based on the operator type and computational characteristics of each layer of the machine learning model, the baseline latency of each basic operator is obtained by querying the measured latency lookup table. Combined with the data dependency relationship between layers, a pipeline parallel overhead correction term is introduced to calculate the overall inference latency of the model.

3. The hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices according to claim 1, characterized in that, The objective function of the multi-objective optimization strategy in step S2 is a weighted combination of minimizing inference latency and maximizing model accuracy. The constraints include that the number of channels after pruning in each layer is not less than a preset lower limit and the sparsity of the weights in each layer is not higher than a preset upper limit.

4. The hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices according to claim 1, characterized in that, The method of utilizing the N-order historical search information in step S2 is as follows: Define the search space for the current iteration as: ， ， in, For the first The model selected in the next iteration. This is a function for evaluating model accuracy. Indicates by the first The iteration model in delayed budget The set of all candidate pruning models generated under constraints.

5. The hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices according to claim 1, characterized in that, The step S2, which involves selecting the Pareto optimal model sequence from the candidate pruning model set, includes: Calculate the inference latency and accuracy of each candidate model in the candidate pruning model set; use inference latency and accuracy as two-dimensional objectives to select non-dominated solutions to form a Pareto front; select models whose accuracy meets a preset accuracy threshold from the Pareto front as the Pareto optimal model sequence.

6. The hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices according to claim 1, characterized in that, The step S3, which calculates the sensitivity of each network layer to the overall accuracy of the model, includes: For each network layer, calculate the change in accuracy of the model on the validation set before and after pruning the layer; divide the change in accuracy by the pruning ratio of the layer parameters to obtain the normalized sensitivity score of the layer; the sensitivity threshold is dynamically determined based on the statistical distribution of the normalized sensitivity scores of all layers.

7. The hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices according to claim 1, characterized in that, The differentiated hierarchical pruning strategy in step S3 further includes: According to the order of the network layers in the forward propagation of the model, each layer is divided into multiple stages; a consistent pruning rate constraint is applied to each layer within the same stage; and the differentiated pruning strategy is adopted between different stages.

8. The hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices according to claim 1, characterized in that, The operator fusion in step S4 includes fusing the convolutional layer, batch normalization layer, and activation function layer into a single computational unit; the memory layout optimization includes rearranging the weight matrix of the compression model according to the memory access characteristics of the target edge device.

9. The hierarchical pruning and compressed inference acceleration method for machine learning models for edge devices according to claim 1, characterized in that, The machine learning model is a convolutional neural network or a Transformer network; the structured pruning is channel pruning, filter pruning, or attention head pruning; the target edge device is an embedded AI development board, a smart collection device, or an IoT terminal device.

Images