An end-side device-oriented lightweight AI algorithm optimization method and system, a terminal, and a storage medium

Abstract

This invention discloses a lightweight AI algorithm optimization method, system, terminal, and storage medium for edge devices, including: calculating the sensitivity score of each layer of the model and setting an adaptive pruning threshold according to the score; removing redundant weights in the model pruning process, performing knowledge distillation using a distillation-warm-up method, generating a sparse matrix storage format, and using NPU sparse operators to accelerate inference; using the INT4 quantization method to quantize the embedding and feedforward network layers of the pruned model, and retaining the FP16 quantization results of the attention score layer; constructing a calibration set based on real TV voice commands, and optimizing the weight quantization according to the calibration set and the KL divergence minimization strategy; packaging the pruning mask, optimized quantization parameters, and inter-layer configuration information into a model package in a preset format, and distributing the packaged model to the terminal device for deployment; this invention improves the accuracy of the lightweight model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence edge computing technology, and in particular to a lightweight AI algorithm optimization method, system, terminal, and storage medium for edge devices. Background Technology

[0002] Traditional lightweight model methods have significant limitations when it comes to lightweight deployment. Regarding model pruning, conventional methods often rely on empirically set global thresholds for structured or unstructured pruning, lacking a fine-grained consideration of the sensitivity of each model layer. This can easily lead to a sharp drop in accuracy when the pruning rate increases, making it difficult to strike a balance between compression efficiency and model performance. In terms of model quantization, while using a uniform INT8 quantization strategy (INT8 quantization is a technique that converts floating-point parameters and activation values ​​in a deep learning model into 8-bit integers to reduce memory usage, improve inference speed, reduce power consumption, and maintain model accuracy as much as possible) can reduce computational and storage overhead to some extent, sensitive modules such as feature extraction layers have low tolerance for quantization errors. A single-precision quantization method can easily lead to a decrease in the expressive power of key features, thus affecting the overall model accuracy.

[0003] Therefore, existing technologies still need improvement. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a lightweight AI algorithm optimization method, system, terminal and storage medium for edge devices, in order to solve the problems of low model accuracy and large damage to the feature extraction layer caused by traditional lightweight AI algorithm deployment methods.

[0005] The technical solution adopted by this invention to solve the technical problem is as follows: In a first aspect, the present invention provides a lightweight AI algorithm optimization method for edge devices, comprising: Calculate the sensitivity score for each layer of the original Transformer model and set an adaptive pruning threshold based on the scores. Redundant weights in the model pruning process are removed in stages. Knowledge distillation is performed between each stage using a distillation and warm-up method, and a sparse matrix storage format is generated. NPU sparse operators are used to accelerate inference. The INT4 quantization method was used to quantize the embedding and feedforward network layers of the pruned model, and the FP16 quantization results of the attention score layer were preserved. A calibration set is constructed based on real voice commands from television, and the weight quantization is optimized according to the calibration set and the KL divergence minimization strategy. The pruned mask and optimized quantization parameter inter-layer configuration information are packaged into a model package with a preset format, and the packaged model is then sent to the terminal device for deployment.

[0006] In one implementation, the step of calculating a sensitivity score for each layer of the original Transformer model and setting an adaptive pruning threshold based on the score ranking includes: A hybrid metric is used to calculate the corresponding sensitivity score for each layer of the original Transformer model. Each layer is arranged from low to high according to the sensitivity score, and the pruning threshold of each layer is dynamically determined based on the sensitivity distribution.

[0007] In one implementation, the step of removing redundant weights during the model pruning process in stages, and performing knowledge distillation using a distillation-warm-up approach between each stage, includes: The pruning process of the original Transformer model is divided into three progressive stages. In each stage, the sensitivity score is re-evaluated and the adaptive threshold is updated based on the current state of the model. The pruning granularity is gradually approached from coarse to fine to approach the target sparsity. A distillation and warm-up mechanism is introduced between each stage. The original Transformer model is used as the teacher model to perform knowledge distillation on the pruned student model. The accuracy loss caused by pruning is repaired by soft label alignment and feature map imitation.

[0008] In one implementation, the generation of the sparse matrix storage format, utilizing NPU sparse operators to accelerate inference, includes: The pruned model weights are converted into a sparse matrix storage format, and the NPU sparse operators are used to accelerate model inference.

[0009] In one implementation, the INT4 quantization method is used to quantize the embedded and feedforward network layers of the pruned model, and the FP16 quantization result of the attention score layer is retained, including: The INT4 quantization method, either symmetric or asymmetric, is used to quantize the embedding of the pruned model and the feedforward network layer, mapping the original FP16 weights or FP32 weights to a preset integer range. The projection weights corresponding to the queries, keys, and values ​​in the attention score layer are all operated on using FP16, as are the dot product results before Softmax in the attention score layer, in order to preserve the FP16 quantization results.

[0010] In one implementation, the step of constructing a calibration set based on real television voice commands and optimizing the weight quantization according to the calibration set and the KL divergence minimization strategy includes: The system collects the actual voice commands from the television and constructs the calibration set based on these commands; wherein the actual voice commands from the television include: channel changing commands, volume adjustment commands, application opening commands, and program search commands. The FP16 model is run on the calibration set, and for the embedding and feedforward network layers to be quantized, the threshold that minimizes the KL divergence is selected as the quantization parameter.

[0011] In one implementation, the step of uniformly packaging the pruned mask and optimized quantization parameter inter-layer configuration information into a model package of a preset format, and then distributing the packaged model to the terminal device for deployment, includes: The pruned mask, optimized quantization parameters of each quantization layer, quantization type of each layer, group size, and end order are packaged into a model package in the preset format and deployed to TV terminals via OTA or application market.

[0012] Secondly, this invention provides a lightweight AI algorithm optimization system for edge devices: The pruning threshold setting module is used to calculate the sensitivity score for each layer of the original Transformer model and set an adaptive pruning threshold according to the score. The distillation and acceleration module is used to remove redundant weights in the model pruning process in stages. Knowledge distillation is performed between each stage using a distillation and warm-up method, and a sparse matrix storage format is generated. The NPU sparse operator is used to accelerate inference. The quantization module is used to quantize the embedded and feedforward network layers of the pruned model using the INT4 quantization method, and to retain the FP16 quantization results of the attention score layer. The quantization parameter optimization module is used to construct a calibration set based on real TV voice commands and optimize the weight quantization according to the calibration set and the KL divergence minimization strategy. The model deployment module is used to package the pruned mask and optimized quantization parameter interlayer configuration information into a model package in a preset format, and then distribute the packaged model to the terminal device for deployment.

[0013] Thirdly, the present invention provides a terminal, including: a processor and a memory, wherein the memory stores a lightweight AI algorithm optimization program for edge devices, and the lightweight AI algorithm optimization program for edge devices, when executed by the processor, is used to implement the operation of the lightweight AI algorithm optimization method for edge devices as described in the first aspect.

[0014] Fourthly, the present invention also provides a computer-readable storage medium storing a lightweight AI algorithm optimization program for edge devices, which, when executed by a processor, is used to implement the operation of the lightweight AI algorithm optimization method for edge devices as described in the first aspect.

[0015] The present invention, by employing the above technical solution, has the following effects: This invention proposes a model compression method that integrates sensitivity-driven progressive pruning and hierarchical hybrid precision quantization. In the pruning stage, by analyzing the sensitivity of model parameters to the final precision layer by layer, connections with high redundancy and low sensitivity are pruned progressively, effectively avoiding a sharp drop in precision and achieving efficient simplification of the model structure. In the quantization stage, a hierarchical hybrid precision strategy is adopted, which configures the bit width differently according to the sensitivity of different layers to quantization errors. High precision is retained for critical layers, while low bit width quantization is used for non-critical layers, thereby minimizing the average bit width of the weights while ensuring the overall accuracy of the model. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the lightweight AI algorithm optimization method for edge devices in this invention.

[0018] Figure 2 This is a flowchart of the progressive pruning process in this invention.

[0019] Figure 3 This is a flowchart of the mixed precision quantization and packaging distribution in this invention.

[0020] Figure 4 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0021] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] Exemplary methods Traditional lightweight model methods have significant limitations when it comes to lightweight deployment. Regarding model pruning, conventional methods often rely on empirically set global thresholds for structured or unstructured pruning, lacking a fine-grained consideration of the sensitivity of each model layer. This can easily lead to a sharp drop in accuracy when the pruning rate increases, making it difficult to strike a balance between compression efficiency and model performance. In terms of model quantization, while using a uniform INT8 quantization strategy (INT8 quantization is a technique that converts floating-point parameters and activation values ​​in a deep learning model into 8-bit integers to reduce memory usage, improve inference speed, reduce power consumption, and maintain model accuracy as much as possible) can reduce computational and storage overhead to some extent, sensitive modules such as feature extraction layers have low tolerance for quantization errors. A single-precision quantization method can easily lead to a decrease in the expressive power of key features, thus affecting the overall model accuracy.

[0024] To address the above technical issues, this invention provides a lightweight AI algorithm optimization method for edge devices. The method includes: calculating the sensitivity score of each layer of the model and setting an adaptive pruning threshold based on the scores; removing redundant weights during the model pruning process, performing knowledge distillation using a distillation-warm-up method, generating a sparse matrix storage format, and accelerating inference using NPU sparse operators; quantizing the embedded and feedforward network layers of the pruned model using the INT4 quantization method, and retaining the FP16 quantization results of the attention score layer; constructing a calibration set based on real television voice commands, and optimizing the weight quantization according to the calibration set and the KL divergence minimization strategy; packaging the pruning mask, optimized quantization parameters, and inter-layer configuration information into a model package in a preset format, and deploying the packaged model to the terminal device. This invention improves the accuracy of the lightweight model.

[0025] like Figure 1 As shown, this embodiment of the invention provides a lightweight AI algorithm optimization method for edge devices, including the following steps: Step S100: Calculate the sensitivity score for each layer of the original Transformer model and set an adaptive pruning threshold according to the score sorting.

[0026] In this embodiment, the large language model requires more than 2 GB of memory to run on the TV side, but the current low-end TVs do not have enough RAM, so they cannot be loaded. In such cases, traditional pruning results in a large loss of accuracy, and single INT8 quantization severely damages the key features of the attention layer, so the deployed language model cannot meet the accuracy requirements of the use case.

[0027] This embodiment uses a TV SoC, NPU accelerator, and lightweight LLM (Large Language Model) engine as the system. Through sensitivity-driven progressive pruning and hierarchical hybrid precision quantization, it effectively avoids a sharp drop in precision, achieves efficient simplification of the model structure, and minimizes the average bit width of the weights while ensuring the overall accuracy of the model.

[0028] In this embodiment, in the progressive pruning technique, a sensitivity score is first calculated for each layer of the Transformer. The sensitivity score is used to quantify the importance of each parameter or each structural unit to the final performance of the model.

[0029] Specifically, in one implementation of this embodiment, step S100 includes the following steps: Step S101: Calculate the corresponding sensitivity score for each layer of the original Transformer model using a hybrid metric, arrange each layer from low to high sensitivity score, and dynamically determine the pruning threshold for each layer based on the sensitivity distribution.

[0030] In this embodiment, a hybrid metric (e.g., gradient metric + activation metric) is used to calculate the sensitivity score for each layer of the original Transformer model. Then, each layer is arranged from low to high sensitivity score, where a low sensitivity score indicates that the weight has little impact on the model output and is a redundant weight, which can be removed during model pruning. Finally, the pruning threshold for each layer is dynamically determined based on the sensitivity distribution. Specifically, on the validation set, pruning is gradually accumulated starting from low sensitivity until the model loss increases by more than a certain threshold (e.g., 0.5%), and this accumulated proportion is the pruning rate of that layer.

[0031] like Figure 1 As shown, this embodiment of the invention provides a lightweight AI algorithm optimization method for edge devices, including the following steps: Step S200 involves removing redundant weights during the model pruning process in stages, performing knowledge distillation between each stage using a distillation-warm-up method, generating a sparse matrix storage format, and using NPU sparse operators to accelerate inference.

[0032] In this embodiment, to avoid irreversible accuracy loss caused by a one-time large-scale pruning, the model performance is restored through progressive sparsity and knowledge distillation. That is, redundant weights in the model pruning process are removed in stages, and knowledge distillation is performed between each stage using a distillation warm-up method.

[0033] Specifically, in one implementation of this embodiment, step S200 includes the following steps: Step S201: The pruning process of the original Transformer model is divided into three stages and executed progressively. In each stage, the sensitivity score is re-evaluated and the adaptive threshold is updated based on the current state of the model. The pruning granularity is gradually approached from coarse to fine to approach the target sparsity. Step S202: A distillation and warm-up mechanism is introduced between each stage. The original Transformer model is used as the teacher model to perform knowledge distillation on the pruned student model. The accuracy loss caused by pruning is repaired by soft label alignment and feature map imitation. Step S203: Convert the pruned model weights into a sparse matrix storage format and use the NPU sparse operator to accelerate model inference.

[0034] In this embodiment, the pruning process of the original Transformer model is performed progressively in three stages: Phase 1: Remove the 10%~20% weights with the lowest sensitivity (relative to the final target sparsity) to maintain the basic integrity of the model structure; Phase 2: Achieve 50%~70% of the target sparsity, and the pruning granularity can be increased to a structured pattern (such as entire rows / columns). Phase 3: Achieve the preset target sparsity (e.g., 80%), at which point the model weights are highly sparse.

[0035] In each of the above stages, the sensitivity score is re-evaluated based on the current state of the model and the adaptive threshold is updated, and the pruning granularity is gradually approached from coarse to fine to approach the target sparsity.

[0036] Meanwhile, a distillation warm-up mechanism is introduced between each stage, using the original Transformer model as the teacher model to perform knowledge distillation on the pruned student model; during the distillation warm-up process, after each pruning stage, a small number of additional training epochs (e.g., 5-10 epochs) are used for distillation fine-tuning to restore the model's expressive ability.

[0037] In this embodiment, to make the output probability distribution of the student model approximate that of the teacher model and retain the class discrimination ability of the original model, the accuracy loss caused by pruning can be repaired by soft label alignment. At the same time, to make the intermediate layer feature representation of the student model align with the teacher model and make up for the information loss caused by pruning, the accuracy loss caused by pruning can also be repaired by feature map imitation. This embodiment guides the sparse student model after pruning to recover its performance through a distillation method that complements soft label alignment and feature map imitation.

[0038] In this embodiment, after pruning, the pruned weight matrix has a large number of zero values, therefore it needs to be converted to an NPU-friendly sparse storage format. Specifically, depending on the NPU hardware characteristics, CSR (Compressed Sparse Row) or its variant Block Sparse CSR storage format can be selected to convert the pruned model weights into a sparse matrix for storage. After converting the pruned model weights into a sparse matrix for storage, NPU sparse operators can be used to accelerate inference in order to improve the model's inference efficiency.

[0039] like Figure 1 As shown, this embodiment of the invention provides a lightweight AI algorithm optimization method for edge devices, including the following steps: In step S300, the INT4 quantization method is used to quantize the embedded and feedforward network layers of the pruned model, and the FP16 quantization results of the attention score layer are retained.

[0040] In this embodiment, after completing the above progressive pruning process, weight quantization optimization is performed using a mixed precision quantization method. This process adopts a mixed precision quantization strategy to significantly reduce the storage and computational overhead of the Embedding layer and the FNN-down layer while ensuring the accuracy of key attention calculations.

[0041] Specifically, in one implementation of this embodiment, step S300 includes the following steps: Step S301: Use symmetric or asymmetric INT4 quantization method to quantize the embedding of the pruned model and the feedforward network layer, and map the original FP16 weights or FP32 weights to a preset integer range. Step S302: The projection weights corresponding to the query, key, and value of the attention score layer are maintained by FP16 operation, and the dot product results before Softmax in the attention score layer are also maintained by FP16 operation to preserve the FP16 quantization results.

[0042] In this embodiment, for the Embedding layer, symmetric or asymmetric INT4 quantization is used to map the original FP16 / FP32 weights to the integer range of [-8, 7] or [0, 15]. Since the number of parameters in the Embedding layer is typically large, INT4 can reduce memory usage by 75% and is robust to activation value distribution. For the FNN-down layer (projection layer under a feedforward network), INT4 quantization is also used. This layer is responsible for feature dimensionality reduction in the speech recognition model and has a high tolerance for quantization noise. Weights need to be grouped and quantized per-channel to mitigate the impact of outliers.

[0043] Furthermore, the FP16 quantization results are retained for the Attention-score layer. Since the Attention-score matrix directly determines the correlation weights between speech frames and is extremely sensitive to recognition accuracy, using INT4 or INT8 quantization may lead to gradient vanishing or distortion of the attention distribution. Therefore, by retaining the FP16 quantization results, the recognition accuracy of the optimized model can be guaranteed.

[0044] As an example, this implementation maintains FP16 computation for the projection weights of the Query, Key, and Value layers in the Attention-score layer, as well as the dot product results before Softmax. Subsequently, these can be concatenated with the Embedding / FNN-down output of INT4 after precision conversion, using a mixed-precision operator of CUDA or TVM. Although FP16 computation is slightly higher than INT4, the number of parameters in the Attention-score layer typically only accounts for 10% to 20% of the total model parameters, thus the overall compression ratio can still reach 3 to 4 times.

[0045] In another implementation of this embodiment, during the quantization process of embedding the pruned model and the feedforward network layer, INT8 quantization can be used, and dynamic quantization compensation can be used to dynamically compensate for the accuracy loss.

[0046] like Figure 1 As shown, this embodiment of the invention provides a lightweight AI algorithm optimization method for edge devices, including the following steps: Step S400: Construct a calibration set based on real voice commands from the television, and optimize the weight quantization according to the calibration set and the KL divergence minimization strategy.

[0047] In this embodiment, after using INT4 quantization and retaining FP16, real TV voice commands are used as the calibration set, and the quantization parameters are optimized with the goal of minimizing KL divergence. Finally, the quantization parameters and sparse mask are packaged into a unified format (.tvlpkg) and sent to the terminal to achieve lightweight deployment.

[0048] Specifically, in one implementation of this embodiment, step S400 includes the following steps: Step S401: Collect the actual TV voice commands and construct the calibration set based on the actual TV voice commands; wherein, the actual TV voice commands include: channel changing commands, volume adjustment commands, application opening commands, and program search commands; Step S402: Run the FP16 model on the calibration set. For the embedding and feedforward network layers to be quantized, select the threshold that minimizes the KL divergence as the quantization parameter.

[0049] In this embodiment, at least 500 voice commands from real-world television usage scenarios are collected, covering typical commands such as "changing channels," "adjusting volume," "opening applications," and "searching for programs," and including background noise (such as the original television sound and air conditioner noise) to match the actual deployment environment. Then, a calibration set is constructed based on the collected real-world television voice commands, and the quantization parameters are optimized with minimizing the KL divergence as the optimization objective.

[0050] In this embodiment, for the embedding and FNN-down layer to be quantized, KL divergence (Kullback-Leibler Divergence) is used as a measure of the difference in output distribution before and after quantization. Specifically: First, run the FP16 model on the calibration set and collect the activation values ​​of the output of each layer.

[0051] Then, iterate through the possible INT4 thresholds (e.g., stepping 0.01 from -7.5 to 7.5) and calculate the KL divergence between the quantized distribution and the original distribution.

[0052] Finally, the threshold that minimizes the KL divergence is chosen as the quantization parameters (scale and zero point).

[0053] For weight quantization, we directly perform KL optimization on the weight values ​​instead of using a simple min-max method.

[0054] Compared to mean squared error (MSE), the KL divergence used in this embodiment can better preserve the information entropy of the original distribution, and is especially suitable for the multi-peak distribution characteristics of voice commands.

[0055] like Figure 1 As shown, this embodiment of the invention provides a lightweight AI algorithm optimization method for edge devices, including the following steps: Step S500: Package the pruning mask and optimized quantization parameter inter-layer configuration information into a model package with a preset format, and then send the packaged model to the terminal device for deployment.

[0056] Specifically, in one implementation of this embodiment, step S500 includes the following steps: Step S501: Package the pruned mask, the optimized quantization parameters of each quantization layer, the quantization type of each layer, the group size, and the end order into a model package of the preset format, and distribute it to the TV terminal for deployment via OTA or application market.

[0057] In this embodiment, the pruned mask, optimized quantization parameters of each quantization layer, quantization type of each layer, group size, end-order, and other data are packaged together. The packaged file format is .tvlpkg, and the specific packaged contents are as follows: 1) Quantization parameters for each quantization layer: scale (float32) and zero_point (int32) for each channel.

[0058] 2) Sparse Mask: Stores the corresponding mask matrix and compresses it using run-length encoding (RLE).

[0059] 3) Model structure metadata: Specifies the quantization type (INT4 / FP16), group size, end order, etc. for each layer.

[0060] In this embodiment, a ".tvlpkg" file can be generated in the cloud and deployed to the TV terminal via OTA or application market; of course, in other implementations of this embodiment, the packaging method of the model is not limited to ".tvlpkg" file, but can also be an open source ONNX file.

[0061] like Figure 2 As shown, in the actual application scenario of this embodiment, the progressive pruning process includes the following steps: Step S10: Calculate the sensitivity score for each layer of the Transformer; Step S20: Sort by score and set an adaptive threshold; Step S30: Redundant weights are removed in three stages, with distillation used for rewarming between stages. Step S40 generates a sparse matrix storage format, which is directly accelerated by the NPU sparse operator.

[0062] like Figure 3 As shown, in the actual application scenario of this embodiment, the mixed precision quantization process includes the following steps: Step S50: INT4 quantization is applied to Embedding and FNN-down. Step S60: Retain the FP16 quantization result for the Attention-score layer; Step S70: Construct a calibration set based on real voice commands from the television and optimize it with the goal of minimizing the KL divergence; Step S80: Package the quantization parameters and sparse mask into a ".tvlpkg" file and send it to the terminal for deployment.

[0063] This invention achieves over 40% sparsity for large language models on a TV SoC platform through sensitivity-driven progressive pruning and hierarchical hybrid precision quantization, with an average weight bit width not exceeding 5 bits, a 55% reduction in inference latency, and Top-1 precision loss controlled within 0.8%. Based on this technological breakthrough, high-end voice assistants can now operate stably offline on budget-friendly TV devices for the first time, providing a viable technical path for edge-side intelligent applications.

[0064] This embodiment achieves the following technical effects through the above technical solution: This invention proposes a model compression method that integrates sensitivity-driven progressive pruning and hierarchical hybrid precision quantization. In the pruning stage, by analyzing the sensitivity of model parameters to the final precision layer by layer, connections with high redundancy and low sensitivity are progressively pruned, effectively avoiding a sharp drop in precision and achieving efficient simplification of the model structure. In the quantization stage, a hierarchical hybrid precision strategy is adopted, configuring bit widths differently according to the sensitivity of different layers to quantization errors. Higher precision is retained for critical layers, while low bit width quantization is used for non-critical layers, thereby minimizing the average bit width of the weights while ensuring the overall accuracy of the model.

[0065] Exemplary device Based on the above embodiments, the present invention also provides a lightweight AI algorithm optimization system for edge devices, comprising: The pruning threshold setting module is used to calculate the sensitivity score for each layer of the original Transformer model and set an adaptive pruning threshold according to the score. The distillation and acceleration module is used to remove redundant weights in the model pruning process in stages. Knowledge distillation is performed between each stage using a distillation and warm-up method, and a sparse matrix storage format is generated. The NPU sparse operator is used to accelerate inference. The quantization module is used to quantize the embedded and feedforward network layers of the pruned model using the INT4 quantization method, and to retain the FP16 quantization results of the attention score layer. The quantization parameter optimization module is used to construct a calibration set based on real TV voice commands and optimize the weight quantization according to the calibration set and the KL divergence minimization strategy. The model deployment module is used to package the pruned mask and optimized quantization parameter interlayer configuration information into a model package in a preset format, and then distribute the packaged model to the terminal device for deployment.

[0066] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 4 As shown.

[0067] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.

[0068] When executed by the processor, this computer program is used to implement a lightweight AI algorithm optimization method for edge devices.

[0069] It will be understood by those skilled in the art that Figure 4 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0070] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a lightweight AI algorithm optimization program for edge devices, the lightweight AI algorithm optimization program for edge devices being executed by the processor to implement the operation of the lightweight AI algorithm optimization method for edge devices as described above.

[0071] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a lightweight AI algorithm optimization program for an edge device, which, when executed by a processor, is used to implement the operation of the above-described lightweight AI algorithm optimization method for an edge device.

[0072] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.

[0073] In summary, this invention provides a lightweight AI algorithm optimization method, system, terminal, and storage medium for edge devices, including: calculating the sensitivity score of each layer of the model and setting an adaptive pruning threshold according to the score; removing redundant weights during the model pruning process, performing knowledge distillation using a distillation-warm-up method, generating a sparse matrix storage format, and using NPU sparse operators to accelerate inference; using the INT4 quantization method to quantize the embedding and feedforward network layers of the pruned model, and retaining the FP16 quantization results of the attention score layer; constructing a calibration set based on real TV voice commands, and optimizing the weight quantization according to the calibration set and the KL divergence minimization strategy; packaging the pruning mask, optimized quantization parameters, and inter-layer configuration information into a model package in a preset format, and distributing the packaged model to the terminal device for deployment; this invention improves the accuracy of the lightweight model.

[0074] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A lightweight AI algorithm optimization method for edge devices, characterized in that, include: Calculate the sensitivity score for each layer of the original Transformer model and set an adaptive pruning threshold based on the scores. Redundant weights in the model pruning process are removed in stages. Knowledge distillation is performed between each stage using a distillation and warm-up method, and a sparse matrix storage format is generated. NPU sparse operators are used to accelerate inference. The INT4 quantization method was used to quantize the embedding and feedforward network layers of the pruned model, and the FP16 quantization results of the attention score layer were preserved. A calibration set is constructed based on real voice commands from television, and the weight quantization is optimized according to the calibration set and the KL divergence minimization strategy. The pruned mask and optimized quantization parameter inter-layer configuration information are packaged into a model package with a preset format, and the packaged model is then sent to the terminal device for deployment.

2. The lightweight AI algorithm optimization method for edge devices according to claim 1, characterized in that, The process of calculating sensitivity scores for each layer of the original Transformer model and setting adaptive pruning thresholds based on these scores includes: A hybrid metric is used to calculate the corresponding sensitivity score for each layer of the original Transformer model. Each layer is arranged from low to high according to the sensitivity score, and the pruning threshold of each layer is dynamically determined based on the sensitivity distribution.

3. The lightweight AI algorithm optimization method for edge devices according to claim 2, characterized in that, The process of removing redundant weights during model pruning in stages, and performing knowledge distillation using a distillation-warm-up method between each stage, includes: The pruning process of the original Transformer model is divided into three progressive stages. In each stage, the sensitivity score is re-evaluated and the adaptive threshold is updated based on the current state of the model. The pruning granularity is gradually approached from coarse to fine to approach the target sparsity. A distillation and warm-up mechanism is introduced between each stage. The original Transformer model is used as the teacher model to perform knowledge distillation on the pruned student model. The accuracy loss caused by pruning is repaired by soft label alignment and feature map imitation.

4. The lightweight AI algorithm optimization method for edge devices according to claim 1, characterized in that, The generated sparse matrix storage format utilizes NPU sparse operators to accelerate inference, including: The pruned model weights are converted into a sparse matrix storage format, and the NPU sparse operators are used to accelerate model inference.

5. The lightweight AI algorithm optimization method for edge devices according to claim 1, characterized in that, The INT4 quantization method is used to quantize the embedded and feedforward network layers of the pruned model, and the FP16 quantization results of the attention score layer are retained, including: The INT4 quantization method, either symmetric or asymmetric, is used to quantize the embedding of the pruned model and the feedforward network layer, mapping the original FP16 weights or FP32 weights to a preset integer range. The projection weights corresponding to the queries, keys, and values ​​in the attention score layer are all operated on using FP16, as are the dot product results before Softmax in the attention score layer, in order to preserve the FP16 quantization results.

6. The lightweight AI algorithm optimization method for edge devices according to claim 1, characterized in that, The process of constructing a calibration set based on real television voice commands and optimizing weight quantization according to the calibration set and the KL divergence minimization strategy includes: The system collects the actual voice commands from the television and constructs the calibration set based on these commands; wherein the actual voice commands from the television include: channel changing commands, volume adjustment commands, application opening commands, and program search commands. The FP16 model is run on the calibration set, and for the embedding and feedforward network layers to be quantized, the threshold that minimizes the KL divergence is selected as the quantization parameter.

7. The lightweight AI algorithm optimization method for edge devices according to claim 1, characterized in that, The process of packaging the pruned mask and optimized quantization parameter inter-layer configuration information into a model package of a preset format, and then distributing the packaged model to the terminal device for deployment, includes: The pruned mask, optimized quantization parameters of each quantization layer, quantization type of each layer, group size, and end order are packaged into a model package in the preset format and deployed to TV terminals via OTA or application market.

8. A lightweight AI algorithm optimization system for edge devices, characterized in that, include: The pruning threshold setting module is used to calculate the sensitivity score for each layer of the original Transformer model and set an adaptive pruning threshold according to the score. The distillation and acceleration module is used to remove redundant weights in the model pruning process in stages. Knowledge distillation is performed between each stage using a distillation and warm-up method, and a sparse matrix storage format is generated. The NPU sparse operator is used to accelerate inference. The quantization module is used to quantize the embedded and feedforward network layers of the pruned model using the INT4 quantization method, and to retain the FP16 quantization results of the attention score layer. The quantization parameter optimization module is used to construct a calibration set based on real TV voice commands and optimize the weight quantization according to the calibration set and the KL divergence minimization strategy. The model deployment module is used to package the pruned mask and optimized quantization parameter interlayer configuration information into a model package in a preset format, and then distribute the packaged model to the terminal device for deployment.

9. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a lightweight AI algorithm optimization program for edge devices, which, when executed by the processor, is used to implement the operation of the lightweight AI algorithm optimization method for edge devices as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a lightweight AI algorithm optimization program for edge devices, which, when executed by a processor, is used to implement the operation of the lightweight AI algorithm optimization method for edge devices as described in any one of claims 1-7.

Images