Dynamic memory compression and tier scheduling method and system based on sparse perception and portable AI device

Abstract

This invention discloses a dynamic memory compression and layer scheduling method, system, and portable AI device based on sparse awareness. The method includes: analyzing the impact of each weight of a trained large model on accuracy, dividing it into key weights and secondary weights and performing asymmetric quantization compression; predicting the access probability of each model layer, performing a prefetch operation in parallel during the current model layer's computation to obtain the predicted subsequent model layer weights after asymmetric quantization compression; storing them in off-chip main memory and completing bus transmission in a compressed state, during which only valid weights are loaded using a pre-generated sparse mask; decompressing the data at the computation entry point; and using a sliding window strategy to dynamically manage the swapping in and out of the model layers: extracting the weight data of the subsequent model layers from the decompressed weight data and moving it into a window in the on-chip main memory, and then sending it to the physical execution engine of the large model to perform inference, thereby achieving stable operation of a large parameter model on a small-capacity memory device.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence reasoning, and in particular to a dynamic memory compression and layer scheduling method, system and portable AI device based on sparse perception in the reasoning process of large language models, which is applicable to the edge side and the end side with limited memory capacity. Background Technology

[0002] Large language model inference requires loading model parameters, key-value caches, and other data into memory. Current mainstream memory management methods mainly include three types: full loading, static paging and swapping, and general memory compression. The full loading method loads the entire model into memory, which requires a very large amount of memory and makes it difficult to run large parameter models on devices with small memory. Static paging swapping swaps out some data to external storage at the layer or block level, resulting in high access latency and poor scheduling flexibility. General memory compression methods use traditional compression algorithms to compress memory data, without making specific optimizations for the sparsity characteristics of large model inference.

[0003] The prior art has at least the following drawbacks: First, the memory capacity bottleneck is prominent: large models with tens of billions or more parameters cannot be run on portable devices with small memory. Secondly, the sparse computation characteristics are not effectively utilized: the KV cache and model parameters have obvious sparse distribution, which existing compression mechanisms cannot detect and optimize. Third, layer scheduling and memory management are decoupled: compression, decompression, prefetching operations and inference computation pipelines have poor coordination, which can easily cause significant waiting delays.

[0004] In summary, existing memory management solutions cannot simultaneously leverage the sparsity of large models, dynamic compression, and layer scheduling to achieve coordinated optimization, making it difficult to achieve low-latency, high-throughput large model inference with limited memory.

[0005] The disclosure of the above background technical content is only for the purpose of assisting in understanding the concept and technical solution of this application, and does not necessarily provide technical instruction. Summary of the Invention

[0006] The purpose of this invention is to provide a dynamic memory compression and layer scheduling method and system based on sparse awareness, which aims to solve the problems of insufficient memory capacity and poor coordination between memory scheduling and compression in large model inference, and to achieve efficient inference of large parameter models under limited memory conditions.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A sparse-aware dynamic memory compression and layer scheduling method includes the following steps: Obtain the weight information of the trained large model, whereby the weight is defined as the strength value of the neurons connected in the large model; The influence of each weight on the model output accuracy is analyzed to divide the weights into key weights and secondary weights. Then, asymmetric quantization compression is performed on the large model weights. A first quantization accuracy is applied to the key weights, and a second quantization accuracy is applied to the secondary weights. The first quantization accuracy is higher than the second quantization accuracy. Sparse feature analysis is performed on the weights, and a sparse mask is generated based on the analyzed sparse features. The positions of weights with non-zero sparse mask are marked as the corresponding valid weights, and the positions of weights with zero sparse mask are marked as the corresponding invalid weights. The access probability of each model layer is predicted to determine the subsequent model layer whose access probability reaches the preset condition. During the calculation of the current model layer, the operation of pre-fetching the weight data of the subsequent model layer is performed in parallel to obtain the data corresponding to the weight of the subsequent model layer after asymmetric quantization compression. The weight data of the subsequent model layer obtained by prefetching is stored in the off-chip main memory and the bus transmission is completed in a compressed state; during the storage of the data in the off-chip main memory, the effective weights are loaded according to the sparse mask after asymmetric quantization compression and stored in the off-chip main memory, or, during the bus transmission, the effective weights are read from the off-chip main memory according to the sparse mask after asymmetric quantization compression. At the computation entry point, the data transmitted on the bus is decompressed by a hardware decompression unit to obtain the decompressed weight data; The following sliding window strategy is used to dynamically manage the swapping in and out of the model layer: after the current model layer has been calculated, the weight data of the current model layer is removed from the on-chip main memory or marked as overwriteable, and the weight data of the subsequent model layer is extracted from the decompressed weight data and stored in the on-chip main memory. The data in the on-chip main memory is configured to be sent to the physical execution engine of the large model to perform inference.

[0008] Furthermore, in accordance with any one or a combination of the aforementioned technical solutions, during the dynamic swapping in and out management of the model layer, the compression status of each memory block is maintained at the granularity of the on-chip main memory, and the compression level of each memory block is dynamically adjusted according to memory usage and / or access status.

[0009] Furthermore, based on any one or a combination of the aforementioned technical solutions, if any of the following triggering conditions are met, the dynamic adjustment of the compression level of each memory block will be triggered: The system's free memory is below the preset memory capacity threshold; Alternatively, the percentage of free memory in the total memory is lower than a preset threshold. Alternatively, the access frequency of a single memory block may vary beyond a preset range; Alternatively, after processing a preset number of tokens, a dynamic adjustment of the compression level of each memory block can be triggered.

[0010] Furthermore, following any one or a combination of the aforementioned technical solutions, if the triggering conditions for dynamically adjusting the compression level of each memory block are met, then the following steps for dynamically adjusting the compression level of each memory block are executed: Update the access statistics for each memory block; If the access frequency of a memory block increases, the current compression level of that memory block will be reduced. If the access frequency of a memory block does not increase, determine whether the system has insufficient free memory. If it does, increase the current compression level of the memory block; otherwise, maintain the current compression level of the memory block. For memory blocks whose compression level has been increased or decreased, the compression status of the memory blocks is readjusted according to the new compression level. The higher the compression level, the higher the compression ratio.

[0011] Furthermore, based on any one or a combination of the aforementioned technical solutions, the initial compression level of each memory block is determined in the following manner: Obtain the prediction result of the access probability of the subsequent model layer to which the data stored in the memory block belongs; If the access probability is greater than the first probability threshold, then the initial compression level of the memory block is determined to be the first compression level; If the access probability is between the first probability threshold and the second probability threshold, then the initial compression level of the memory block is determined to be the second compression level, where the second probability threshold is less than the first probability threshold. If the access probability is lower than the second probability threshold, then the initial compression level of the memory block is determined to be the third compression level; The compression ratio corresponding to the first compression level is higher than that of the second compression level, and the compression ratio corresponding to the second compression level is higher than that of the third compression level.

[0012] Furthermore, following any one or a combination of the aforementioned technical solutions, the impact of each weight on the model output accuracy is analyzed in the following manner: Extract a batch of sample data from the training set or validation set; Input the batch of sample data into the large model and calculate the loss; Calculate the gradient of the loss with respect to each weight; The absolute value of the product of the gradient and the weight, or the absolute value of the gradient, is used as an indicator of importance score. Calculate the importance score of all weights and normalize the results by layer or globally; Based on the ranking of the normalized importance scores from high to low, key weights and secondary weights are divided, wherein the ranking of the key weights is higher than that of the secondary weights.

[0013] Furthermore, based on any one or a combination of the aforementioned technical solutions, the sparse mask is obtained by combining sparse feature analysis and pruning, including: Analyze the impact of each weight on the model output accuracy to obtain the importance score corresponding to each weight; Construct a dimension-aligned weight magnitude matrix and an importance score matrix, wherein the weight values ​​in the weight magnitude matrix are normalized and the importance scores in the importance score matrix are normalized. Correlation analysis was performed on the weight magnitude matrix and importance score matrix to obtain the correlation coefficient; If the correlation coefficient reaches the preset strong correlation threshold, then according to the magnitude of the weight, the weights with the preset pruning ratio are taken from the last position to the front, and the sparse mask of the corresponding weight position is set to 0, while the sparse mask of other weight positions is set to 1. If the correlation coefficient is less than the preset strong correlation threshold, the comprehensive score S for each weight is calculated using the following formula: S = α × W norm +(1-α)×I norm Among them, W norm I represents the normalized magnitude of the weight. norm This represents the normalized importance score of the weight; α is a proportional parameter between 0 and 1, which is set according to the calculated value of the weight pruning risk. The greater the weight pruning risk, the closer the set value of α is to 0, and the smaller the weight pruning risk, the closer the set value of α is to 1. Based on the comprehensive score, the weights with a preset pruning ratio are taken from the last position to the front, and the sparse mask of the corresponding weight position is set to 0, while the sparse mask of other weight positions is set to 1.

[0014] Furthermore, following any one or a combination of the aforementioned technical solutions, the two-dimensional space of weight amplitude and importance score is divided into four quadrants using the median line. The proportion of weights in the safe quadrant region with small weight amplitudes and low importance scores is determined, and the weight pruning risk in the safe quadrant region is quantified. This includes calculating the risk factor Risk for each weight in the safe quadrant region using the following formula: Risk = 1 - [(1 - W norm )×(1-I norm )], W norm I represents the normalized magnitude of the weight. normThe normalized importance score of the weight is used to calculate the average value of the risk factors of all weights within the safety quadrant as the weight pruning risk. Based on the weight percentage in the safety quadrant region, the pruning ratio is set to be less than the weight percentage in the safety quadrant region.

[0015] Furthermore, following any one or a combination of the aforementioned technical solutions, the large model is a Transformer model, and the subsequent model layers are determined in the following manner: Define the current model layer as the nth model layer, predict at least the (n+1)th model layer and the (n+2)th model layer as subsequent model layers, and predict the access probability of the (n+1)th model layer as a preset first access probability value, and predict the access probability of the (n+2)th model layer as a preset second access probability value, wherein the first access probability value is greater than the second access probability value.

[0016] Furthermore, following any one or a combination of the aforementioned technical solutions, the large model is a hybrid expert model or a conditional branching model, and the subsequent model layers are determined based on access history and contextual features: A pre-trained 2-layer MLP model outputs the probability distribution of each model layer based on the temporal features, content features, and load features corresponding to the current model layer. The temporal features include the historical path between model layers, the content features include the semantic and length information of the token, and the load features include system resources and task type information. The top few model layers, sorted by probability value, are used as the subsequent model layers for prediction. Other model layers, except for the predicted subsequent model layers, are stored using a preset high compression ratio. If the actual subsequent model layer is inconsistent with the predicted subsequent model layer, then the weight parameters of the MLP model are optimized.

[0017] Furthermore, following any one or a combination of the aforementioned technical solutions, at the computational entry point, the hardware decompression unit performs real-time decompression, and the decompression operation is executed in parallel with the computational pipeline of the current model layer.

[0018] Furthermore, in accordance with any or a combination of the aforementioned technical solutions, during the process of dynamically managing the model layers using a sliding window strategy, several resident model layers are also configured, and the on-chip main memory continuously stores the data of the resident model layers after asymmetric quantization compression.

[0019] According to another aspect of the present invention, a sparse-aware dynamic memory compression and layer scheduling system is provided, comprising the following modules: The weight file module is configured to store the weight information of a trained large model, wherein the weights are defined as the strength values ​​of the neurons connected in the large model. The importance analysis module is configured to analyze the degree of influence of each weight on the accuracy of the model output, so as to divide the weights into key weights and secondary weights. An asymmetric quantization compression module is configured to perform asymmetric quantization compression on large model weights, wherein a first quantization precision is applied to the key weights and a second quantization precision is applied to the minor weights, and the first quantization precision is higher than the second quantization precision. A sparse mask generation module is configured to perform sparse feature analysis on the weights and generate a sparse mask based on the analyzed sparse features. The positions of weights with non-zero values ​​in the sparse mask are marked as corresponding valid weights, and the positions of weights with zero values ​​in the sparse mask are marked as corresponding invalid weights. The prediction and prefetching module is configured to predict the access probability of each model layer in order to determine the subsequent model layer whose access probability reaches the preset condition. During the calculation of the current model layer, the module performs the operation of prefetching the weight data of the subsequent model layer in parallel to obtain the data corresponding to the weight after asymmetric quantization compression. An off-chip storage module is configured to store the prefetched weight data of subsequent model layers into off-chip main memory; The bus transmission module is configured to transmit data stored in the off-chip main memory in a compressed state, and during the bus transmission process, read the data after asymmetric quantization compression of the effective weights from the off-chip main memory according to the sparse mask, or during the data storage process of the off-chip storage module, load the data after asymmetric quantization compression of the effective weights according to the sparse mask and store it in the off-chip main memory. The decompression module is configured to decompress the data transmitted on the bus at the calculation entry point using a hardware decompression unit to obtain the decompressed weight data. The dynamic layer scheduling module is configured to manage the dynamic swapping in and out of model layers using the following sliding window strategy: after the current model layer has been calculated, the weight data of the current model layer is removed from the on-chip main memory or marked as overwriteable, and the weight data of the subsequent model layer is extracted from the decompressed weight data and stored in the on-chip main memory. The data in the on-chip main memory is configured to be sent to the physical execution engine of the large model for inference.

[0020] Furthermore, following any or a combination of the aforementioned technical solutions, the sparse-aware dynamic memory compression and layer scheduling system also includes a dynamic memory pool unit, which is configured to maintain the compression status, sparsity, and access frequency of each data block at the memory block granularity, and dynamically adjust the compression level. And / or, the system supports the simultaneous deployment and scheduling of multiple large model instances on a single end device.

[0021] According to another aspect of the present invention, a portable AI device is provided, comprising the dynamic memory compression and layer scheduling system as described above, or performing the dynamic memory compression and layer scheduling method as described above.

[0022] The beneficial effects of the technical solution provided by this invention are as follows: a. It addresses the issues of insufficient memory capacity and poor coordination between memory scheduling and compression in large model inference, enabling efficient inference of large parameter models under limited memory conditions. It has stronger adaptability to sparse structure models (such as the MoE model) and more prominent memory optimization effects. b. Combining the importance and sparsity of weights, key weights are quantized with high precision and secondary weights are quantized with low bit depth, achieving sparsity-aware asymmetric quantization compression of large model parameters, significantly reducing memory usage, and enabling large models with large parameters to run on devices with small memory capacity. c. Based on sparse masking, only valid weights are loaded, reducing invalid bus transmissions. Combined with predictive prefetching, memory bandwidth utilization is greatly improved, further enhancing energy efficiency. d. Adopt the "resident layer k + sliding window w" strategy to dynamically swap in and out model layers, and deeply link with memory compression status to achieve the optimal balance between memory and computation, improve inference throughput, and reduce performance jitter caused by memory swapping in and out; e. Hardware-level decompression units hide decompression latency within the computation pipeline, adding almost no additional inference overhead. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a sparse-aware dynamic memory compression and layer scheduling method provided as an exemplary embodiment of the present invention; Figure 2 A flowchart illustrating the influence of each weight on the accuracy of the model output, provided as an exemplary embodiment of the present invention; Figure 3 A flowchart illustrating the process of obtaining the sparse mask by combining sparse feature analysis and pruning, which is an exemplary embodiment of the present invention. Figure 4A schematic diagram of the safe quadrant region in a two-dimensional space of weight magnitude-importance score provided for an exemplary embodiment of the present invention; Figure 5 A schematic diagram of a process for implementing a differentiated layered compression strategy is provided as an exemplary embodiment of the present invention; Figure 6 A flowchart illustrating the dynamic adjustment of compression levels for each memory block, provided as an exemplary embodiment of the present invention; Figure 7 A schematic block diagram of a sparse-aware dynamic memory compression and layer scheduling system provided as an exemplary embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0027] In one embodiment of the present invention, a dynamic memory compression and layer scheduling method based on sparsity awareness is provided to enable large model inference under limited memory conditions, such as... Figure 1 As shown, the dynamic memory compression and layer scheduling method includes the following steps: S100: Obtain the weight information of the trained large model, wherein the weight is defined as the strength value of the neurons connected in the large model; Large models refer to deep neural networks, which typically have a weight data volume that exceeds the on-chip storage capacity. In this embodiment, a large model has no fewer than 24 model layers.

[0028] In the model weight repository, read the weight file of the large language model that has been trained. The weight file includes the weight matrix of each model layer and is stored in a predefined serialization format.

[0029] S200: Analyze the impact of each weight on the model output accuracy to classify the weights into key weights and secondary weights. Then, perform asymmetric quantization compression on the large model weights. A first quantization precision is applied to the key weights, and a second quantization precision is applied to the secondary weights. The first quantization precision is higher than the second quantization precision. In a specific embodiment, the key weights are subjected to 8-bit integer (INT8) asymmetric uniform quantization, where the quantization parameters include a scaling factor and a zero-point, and the quantized integer range is 0 to 255. The secondary weights are subjected to 4-bit integer (INT4) asymmetric uniform quantization, where the quantization parameters include a scaling factor and a zero-point, and the quantized integer range is 0 to 15.

[0030] The degree to which a weight affects the model's output accuracy is a measure of its importance. Importance depends on its contribution (sensitivity) to the final output error, and is not directly related to the absolute value of the weight itself. In this embodiment, importance is defined as the sensitivity of the loss function to changes in that weight. Figure 2 As shown, the impact of each weight on the model output accuracy is analyzed through the following steps: S210: Extract a batch (e.g., 512 or 1024) of sample data from the training or validation set, without labels, only forward propagation is required; S220: Input the batch of sample data into the large model and calculate the loss value. L For models that only perform inference, a simple proxy loss can be constructed, such as the entropy of the output distribution.

[0031] S230: Calculate the gradient of the loss with respect to each weight. g i =∂ L / ∂ w i ,in, g i For the first i The gradient of each weight, w i For the first i The weight value of each weight. L This is the loss value.

[0032] S240: Utilizing the absolute value of the product of the gradient and the weights S i = |gi · w i | or the absolute value of the gradient S i = |g i | As an indicator of importance score; S250: Calculate the importance score for each weight. S i The calculation results are normalized either layer-wise or globally. If normalization is performed layer-wise, the maximum value among the importance scores of multiple weights within the same layer is defined as... S max The minimum value is defined as S min For the first i The importance scores after weight normalization are: S i norm =( S i - S min ) / ( S max - S min In certain embodiments, outliers (importance scores) can be identified before normalization. S i (Abnormally large or abnormally small values) are removed. The same principle applies globally, so I will not repeat it here.

[0033] S260: Based on the ranking of importance scores from high to low after normalization, key weights and secondary weights are divided by layer or globally. For example, if the weights are set to account for 30% of key weights and 70% of secondary weights, then the top 30% of importance scores from high to low are taken as key weights, and the rest are taken as secondary weights.

[0034] S300: Perform sparse feature analysis on the weights and generate a sparse mask based on the analyzed sparse features. The positions of weights with non-zero sparse masks are marked as the corresponding valid weights, and the positions of weights with zero sparse masks are marked as the corresponding invalid weights. There are various ways to generate the mask, such as the threshold pruning method: set the sparse mask corresponding to the weights whose absolute value is less than a preset weight threshold (e.g., 0.02) to 0, and set the sparse mask corresponding to other weights to 1.

[0035] In one embodiment of the present invention, the sparse mask is obtained by combining sparse feature analysis and pruning, as shown below. Figure 3 As shown: S310: Analyze the impact of each weight on the model output accuracy to obtain the importance score corresponding to each weight (see the importance score in step S250). S i ); S320: Construct a dimension-aligned weight magnitude matrix and an importance score matrix, wherein the weight values ​​in the weight magnitude matrix are normalized and the importance scores in the importance score matrix are normalized. Thus, two comparable, normalized feature matrices establish a reference coordinate system for subsequent analysis. S330: Perform correlation analysis on the weight magnitude matrix and importance score matrix, for example, using the Pearson correlation analysis method to obtain the Pearson correlation coefficient; for example, if the Pearson correlation coefficient is 0.72, which is greater than the preset strong correlation threshold of 0.6, then proceed to step S340; otherwise, proceed to step S350. S340: Sort by weight magnitude, take the weights with a preset pruning ratio from the last to the first, set the sparse mask of the corresponding weight position to 0, and set the sparse mask of other weight positions to 1; Since the Pearson correlation coefficient reaches the preset strong correlation threshold, it is equivalent to α being set to 1 in the comprehensive score formula in step S350, indicating that it depends entirely on the weight magnitude. S350: Calculate the overall score for each weight using the following formula: S = α × W norm +(1-α)×I norm Among them, W norm I represents the normalized magnitude of the weight. norm This represents the normalized importance score of the weight; α is a proportional parameter between 0 and 1, which is set according to the calculated value of the weight pruning risk. The greater the weight pruning risk, the closer the set value of α is to 0, and the smaller the weight pruning risk, the closer the set value of α is to 1. S360: Sort by the size of the comprehensive score, take the weight of the preset pruning ratio from the last to the front, set the sparse mask of the corresponding weight position to 0, and set the sparse mask of other weight positions to 1.

[0036] The pruning ratios in steps S340 and S360 above are determined as follows: The median of all weight magnitudes and the median of importance scores are determined, and the two-dimensional space of weight magnitude-importance scores is divided into four quadrants using the median line, such as... Figure 4As shown, with "weight magnitude" as the horizontal axis and "importance score" as the vertical axis, each weight is plotted as a point: weights with a magnitude greater than the median weight magnitude and an importance score greater than the median importance score are located in the upper right quadrant (first quadrant); weights with a magnitude less than the median weight magnitude and an importance score greater than the median importance score are located in the upper left quadrant (second quadrant); weights with a magnitude less than the median weight magnitude and an importance score less than the median importance score are located in the lower left quadrant (third quadrant, also known as the safety quadrant); and weights with a magnitude greater than the median weight magnitude and an importance score less than the median importance score are located in the lower right quadrant (fourth quadrant). The percentage of weights in the safety quadrant is determined, and the pruning ratio is set to be less than the percentage of weights in the safety quadrant. The weight pruning risk in step S350 above is quantified as follows: The risk factor Risk for each weight within the safety quadrant is calculated using the following formula: Risk = 1 - [(1 - W norm )×(1-I norm )], W norm I represents the normalized magnitude of the weight. norm The normalized importance score of the weight is represented; the average value of the risk factors of all weights in the safety quadrant is calculated as the weight pruning risk. The greater the weight pruning risk, the closer the value of α in the comprehensive score formula in step S350 is to 0. The smaller the weight pruning risk, the closer the value of α is to 1.

[0037] S400: Perform access probability prediction on each model layer to determine the subsequent model layer whose access probability reaches the preset condition; Specifically, the specific method for predicting access probabilities is closely related to the type of large model: Scenario 1: The large model is a Transformer model, and the subsequent model layers are determined in the following way: Define the current model layer as the nth model layer, predict at least the (n+1)th model layer and the (n+2)th model layer as subsequent model layers, and predict the access probability of the (n+1)th model layer as a preset first access probability value, and predict the access probability of the (n+2)th model layer as a preset second access probability value, wherein the first access probability value is greater than the second access probability value.

[0038] Scenario 2: The large model is a hybrid expert model (MoE) or a conditional branching model, and the subsequent model layers are determined based on access history and contextual features. A pre-trained 2-layer MLP model outputs the probability distribution of each model layer based on the temporal features, content features, and load features corresponding to the current model layer. The temporal features include the historical path between model layers, the content features include the semantic and length information of the token, and the load features include system resources and task type information. The top few model layers, sorted by probability value, are used as the subsequent model layers for prediction. Other model layers, except for the predicted subsequent model layers, are stored using a preset high compression ratio. If the actual subsequent model layer is inconsistent with the predicted subsequent model layer, then the weight parameters of the MLP model are optimized.

[0039] This invention does not limit the execution order of steps S300 and S400, such as Figure 1 As shown, step S300 serves step S600 below, and steps S400 and S200 together serve step S500 below.

[0040] S500: During the calculation of the current model layer, the operation of prefetching the weight data of the subsequent model layer is performed in parallel to obtain the data corresponding to the weight of the subsequent model layer after asymmetric quantization compression. S600: Store the prefetched weight data of the subsequent model layer into off-chip main memory and complete the bus transmission in compressed state; Step S300 serves Step S600, which means loading only the effective weights according to the sparse mask. It can be applied in either of the following two cases: Case 1: During the storage of data into off-chip main memory, the effective weights are loaded according to the sparse mask after asymmetric quantization compression and stored into off-chip main memory, which can save memory resources; Case 2: During the bus transmission, the effective weights are read from the off-chip main memory according to the sparse mask after asymmetric quantization compression, avoiding zero-mask data from occupying bus bandwidth.

[0041] S700: At the computation entry point, the data transmitted on the bus is decompressed by a hardware decompression unit to obtain the decompressed weight data. Specifically, the hardware decompression unit is a dedicated hardware acceleration circuit for real-time decompression. Real-time decompression is performed at the computation entry point. The decompression operation is executed in parallel with the computation pipeline of the current model layer, so no significant additional overhead is generated.

[0042] S800: The following sliding window strategy is used to dynamically manage the swapping in and out of model layers: After the current model layer is calculated, the weight data of the current model layer is removed from the on-chip main memory or marked as overwriteable, and the weight data of the subsequent model layer is extracted from the decompressed weight data and stored in the on-chip main memory. The data in the on-chip main memory is configured to be sent to the physical execution engine (such as dNPU) of the large model to perform inference.

[0043] The on-chip main memory in step S800 and the off-chip main memory in step S600 are different storage areas. Off-chip main memory refers to external DRAM, which is used for compressed storage and bus transmission. On-chip main memory refers to on-chip SRAM, which is used for dynamic swap-in and swap-out management. Typical system-on-chip (SoC) includes on-chip SRAM / Cache and off-chip DRAM. Based on the hierarchical storage architecture composed of on-chip system memory and off-chip main memory, this invention stores the weight data of subsequent model layers obtained by prefetching into the off-chip dynamic random access memory (DRAM). At the calculation entry point, the data decompressed by the hardware decompression unit is transferred from DRAM to on-chip static random access memory (SRAM). The "resident layer k + sliding window w" strategy can be used to dynamically swap in and out of SRAM: that is, load k resident layers (such as layers 0-2 and the last layer that are basically used in each inference) into a fixed area of ​​SRAM, and determine the number of layers w that the sliding window can accommodate based on the remaining capacity of SRAM and the weight of a single layer. The current computation layer is defined as L. The sliding window contains the weights of layer [L, L+1, …, L+w-1]. When layer L is computed, prefetching is triggered: the weights of the subsequent model layer L+w are loaded from DRAM into the SRAM buffer. When layer L is computed, the window advances: the weights of layer L are marked as overwriteable (or deleted), and the weight data of layer L+w is moved from the buffer into the active window. Only at the moment the hardware instruction "load the weights of layer j" is issued can the subsequent model layer be determined with 100% certainty. If the prediction of the subsequent model layer is correct, the layer data in the sliding window w is sent layer by layer to the computation unit (e.g., dNPU) to perform large model inference computation. If the prediction is wrong, that is, the predicted subsequent layer does not match the hardware instruction, then the layer data is processed according to the normal layer data. This fallback mechanism ensures adaptability, avoids pipeline stalls, and enhances the robustness of the solution.

[0044] In one embodiment, dynamic swapping in and out of the model layer is performed concurrently with dynamic memory management: the compression status of each memory block is maintained at the granularity of the on-chip main memory, and the compression level of each memory block is dynamically adjusted based on memory usage and / or access patterns; for large MoE architecture models where only some experts are activated, parameter blocks corresponding to inactive experts are marked as cold blocks and stored with high compression; data of experts to be activated is pre-fetched based on routing prediction results; memory usage is significantly reduced, supporting the simultaneous deployment of multiple model instances on a single device. If any of the following trigger conditions are met, the compression level of each memory block will be dynamically adjusted: The system's free memory is below the preset memory capacity threshold; Alternatively, the percentage of system free memory in total memory is lower than a preset threshold (e.g., 10%). Alternatively, the frequency of access to a single memory block may vary beyond a preset range (e.g., 50%). Alternatively, after processing a preset number (e.g., 100) of tokens, a dynamic adjustment of the compression level of each memory block can be triggered.

[0045] If the triggering conditions for dynamically adjusting the compression level of each memory block are met, then the following steps for dynamically adjusting the compression level of each memory block will be executed: Update the access statistics for each memory block; If the access frequency of a memory block increases, the current compression level of that memory block is reduced, and the compression ratio is reduced accordingly to improve access speed. If the access frequency of a memory block does not increase, determine whether the system has insufficient free memory. If it does, increase the current compression level of the memory block and increase the compression ratio accordingly to save memory space; otherwise, maintain the current compression level of the memory block. For memory blocks whose compression level has been increased or decreased, the compression status of the memory blocks is readjusted according to the new compression level; the higher the compression level, the higher the compression ratio. In a particular embodiment, the compression strategy can adaptively adjust the corresponding compression ratio based on the actual inference scenario, model accuracy requirements, and memory capacity.

[0046] Specific numerical implementation example: Memory blocks are divided into hot blocks, warm blocks, and cold blocks based on access probability, and a differentiated layered compression strategy is executed, such as... Figure 5 As shown, the initial compression level of each memory block is determined in the following way: Obtain the prediction result of the access probability of the subsequent model layer to which the data stored in the memory block belongs; If the access probability is greater than the first probability threshold (0.8), then the initial compression level of the memory block is determined to be the first compression level, that is, it is divided into a hot block and is subjected to no compression (compression ratio of 1:1) or light compression (compression ratio of 1:1.5). If the access probability is between the first probability threshold (0.8) and the second probability threshold (0.3), then the initial compression level of the memory block is determined to be the second compression level, that is, it is divided into a warm block and medium compression is adopted (compression ratio between 1:2 and 1:4). If the access probability is lower than the second probability threshold (0.3), the initial compression level of the memory block is determined to be the third compression level, that is, it is classified as a cold block and highly compressed (compression ratio between 1:5 and 1:10).

[0047] like Figure 6As shown, if the triggering conditions for dynamically adjusting the compression level of each memory block are met, and the access frequency of the corresponding memory block increases, the current compression level of the memory block will be reduced: if the current memory block is a hot block, it will be downgraded to a warm block, and the compression ratio will be reduced to 1:2 to 1:4; if the current memory block is a warm block, it will be downgraded to a cold block, and the compression ratio will be reduced to 1:5 to 1:10.

[0048] If the triggering conditions for dynamically adjusting the compression level of each memory block are met, and the access frequency of the corresponding memory block does not increase, then it is further determined whether the system's free memory is less than the preset free capacity threshold. If it is less, the current compression level of the memory block is increased: if the current memory block is a warm block, it is upgraded to a hot block, and the compression ratio is reduced to 1:1 to 1:1.5; if the current memory block is a cold block, it is upgraded to a warm block, and the compression ratio is reduced to 1:2 to 1:4. If the system's free memory reaches the preset free capacity threshold, the current compression level (compression ratio) remains unchanged.

[0049] The sparse-aware dynamic memory compression and layer scheduling method provided in the above embodiments is applicable to edge devices, mobile AI devices with limited memory, and can complete large-parameter large-model inference under limited memory resources.

[0050] On edge devices with limited memory, the weights of large models are classified by importance and compressed asymmetrically; sparse masks are generated, only valid weights are loaded and data for subsequent layers is prefetched; the model layers are dynamically scheduled using a "resident layer k + sliding window w" strategy, and the hardware real-time decompression unit hides the decompression delay in the computation pipeline; finally, the large parameter model runs stably under limited memory capacity, maintaining high inference throughput.

[0051] This invention also provides a sparse-aware dynamic memory compression and layer scheduling system, such as... Figure 7 As shown, the system includes the following modules: The weight file module is configured to store the weight information of a trained large model, wherein the weights are defined as the strength values ​​of the neurons connected in the large model. The importance analysis module is configured to analyze the degree of influence of each weight on the accuracy of the model output, so as to divide the weights into key weights and secondary weights. An asymmetric quantization compression module is configured to perform asymmetric quantization compression on large model weights, wherein a first quantization precision is applied to the key weights and a second quantization precision is applied to the minor weights, and the first quantization precision is higher than the second quantization precision. A sparse mask generation module is configured to perform sparse feature analysis on the weights and generate a sparse mask based on the analyzed sparse features. The positions of weights with non-zero values ​​in the sparse mask are marked as corresponding valid weights, and the positions of weights with zero values ​​in the sparse mask are marked as corresponding invalid weights. The prediction and prefetching module is configured to predict the access probability of each model layer in order to determine the subsequent model layer whose access probability reaches the preset condition. During the calculation of the current model layer, the module performs the operation of prefetching the weight data of the subsequent model layer in parallel to obtain the data corresponding to the weight after asymmetric quantization compression. An off-chip storage module is configured to store the prefetched weight data of subsequent model layers into off-chip main memory; The bus transmission module is configured to transmit data stored in the off-chip main memory in a compressed state, and during the bus transmission process, read the data after asymmetric quantization compression of the effective weights from the off-chip main memory according to the sparse mask, or during the data storage process of the off-chip storage module, load the data after asymmetric quantization compression of the effective weights according to the sparse mask and store it in the off-chip main memory. The decompression module is configured to decompress the data transmitted on the bus at the calculation entry point using a hardware decompression unit to obtain the decompressed weight data. The dynamic layer scheduling module is configured to manage the dynamic swapping in and out of model layers using the following sliding window strategy: after the current model layer has been calculated, the weight data of the current model layer is removed from the on-chip main memory or marked as overwriteable, and the weight data of the subsequent model layer is extracted from the decompressed weight data and stored in the on-chip main memory. The data in the on-chip main memory is configured to be sent to the physical execution engine of the large model for inference.

[0052] Furthermore, following any one or a combination of the aforementioned technical solutions, the sparse-aware dynamic memory compression and layer scheduling system also includes a dynamic memory pool unit, which is configured to maintain the compression status, sparsity and access frequency of each data block at the memory block granularity, and dynamically adjust the compression level, maintain the sparse mask of the large model weight, the access frequency of each model layer and dynamically adjust the compression level of each memory block of the on-chip main memory. And / or, the system supports the simultaneous deployment and scheduling of multiple large model instances on a single end device.

[0053] It should be noted that the sparse-aware dynamic memory compression and layer scheduling system provided in this embodiment and the sparse-aware dynamic memory compression and layer scheduling method provided in the above embodiments belong to the same inventive concept. Here, the entire contents of the embodiment of the sparse-aware dynamic memory compression and layer scheduling method are incorporated into this embodiment of the sparse-aware dynamic memory compression and layer scheduling system by reference, and will not be repeated.

[0054] Accordingly, embodiments of the present invention provide a portable AI device, including the dynamic memory compression and layer scheduling system described above, or executing the dynamic memory compression and layer scheduling method described above.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0056] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A dynamic memory compression and layer scheduling method based on sparse awareness, characterized in that, Includes the following steps: Obtain the weight information of the trained large model, whereby the weight is defined as the strength value of the neurons connected in the large model; The influence of each weight on the model output accuracy is analyzed to divide the weights into key weights and secondary weights. Then, asymmetric quantization compression is performed on the large model weights. A first quantization accuracy is applied to the key weights, and a second quantization accuracy is applied to the secondary weights. The first quantization accuracy is higher than the second quantization accuracy. Sparse feature analysis is performed on the weights, and a sparse mask is generated based on the analyzed sparse features. The positions of weights with non-zero sparse mask are marked as the corresponding valid weights, and the positions of weights with zero sparse mask are marked as the corresponding invalid weights. The access probability of each model layer is predicted to determine the subsequent model layer whose access probability reaches the preset condition. During the calculation of the current model layer, the operation of pre-fetching the weight data of the subsequent model layer is performed in parallel to obtain the data corresponding to the weight of the subsequent model layer after asymmetric quantization compression. The weight data of the subsequent model layer obtained by prefetching is stored in the off-chip main memory and the bus transmission is completed in a compressed state; during the storage of the data in the off-chip main memory, the effective weights are loaded according to the sparse mask after asymmetric quantization compression and stored in the off-chip main memory, or, during the bus transmission, the effective weights are read from the off-chip main memory according to the sparse mask after asymmetric quantization compression. At the computation entry point, the data transmitted on the bus is decompressed by a hardware decompression unit to obtain the decompressed weight data; The following sliding window strategy is used to dynamically manage the swapping in and out of the model layer: after the current model layer has been calculated, the weight data of the current model layer is removed from the on-chip main memory or marked as overwriteable, and the weight data of the subsequent model layer is extracted from the decompressed weight data and stored in the on-chip main memory. The data in the on-chip main memory is configured to be sent to the physical execution engine of the large model to perform inference.

2. The dynamic memory compression and layer scheduling method according to claim 1, characterized in that, During the dynamic swapping in and out management of the model layer, the compression status of each memory block is maintained at the granularity of the on-chip main memory, and the compression level of each memory block is dynamically adjusted according to memory usage and / or access status.

3. The dynamic memory compression and layer scheduling method according to claim 2, characterized in that, If any of the following trigger conditions are met, the compression level of each memory block will be dynamically adjusted: The system's free memory is below the preset memory capacity threshold; Alternatively, the percentage of system free memory in total memory is lower than a preset threshold. Alternatively, the access frequency of a single memory block may vary beyond a preset range; Alternatively, after processing a preset number of tokens, a dynamic adjustment of the compression level of each memory block can be triggered.

4. The dynamic memory compression and layer scheduling method according to claim 2, characterized in that, If the triggering conditions for dynamically adjusting the compression level of each memory block are met, then the following steps for dynamically adjusting the compression level of each memory block will be executed: Update the access statistics for each memory block; If the access frequency of a memory block increases, the current compression level of that memory block will be reduced. If the access frequency of a memory block does not increase, determine whether the system has insufficient free memory. If it does, increase the current compression level of the memory block. Otherwise, maintain the current compression level of the memory block; For memory blocks whose compression level has been increased or decreased, the compression status of the memory blocks is readjusted according to the new compression level. The higher the compression level, the higher the compression ratio.

5. The dynamic memory compression and layer scheduling method according to claim 2, characterized in that, The initial compression level of each memory block is determined in the following way: Obtain the prediction result of the access probability of the subsequent model layer to which the data stored in the memory block belongs; If the access probability is greater than the first probability threshold, then the initial compression level of the memory block is determined to be the first compression level; If the access probability is between the first probability threshold and the second probability threshold, then the initial compression level of the memory block is determined to be the second compression level, where the second probability threshold is less than the first probability threshold. If the access probability is lower than the second probability threshold, then the initial compression level of the memory block is determined to be the third compression level; The compression ratio corresponding to the first compression level is higher than that of the second compression level, and the compression ratio corresponding to the second compression level is higher than that of the third compression level.

6. The dynamic memory compression and layer scheduling method according to claim 1, characterized in that, The impact of each weight on the model output accuracy is analyzed using the following methods: Extract a batch of sample data from the training set or validation set; Input the batch of sample data into the large model and calculate the loss; Calculate the gradient of the loss with respect to each weight; The absolute value of the product of the gradient and the weight, or the absolute value of the gradient, is used as an indicator of importance score. Calculate the importance score of all weights and normalize the results by layer or globally; Based on the ranking of the normalized importance scores from high to low, key weights and secondary weights are divided, wherein the ranking of the key weights is higher than that of the secondary weights.

7. The dynamic memory compression and layer scheduling method according to claim 1, characterized in that, The sparse mask is obtained by combining sparse feature analysis and pruning, including: Analyze the impact of each weight on the model output accuracy to obtain the importance score corresponding to each weight; Construct a dimension-aligned weight magnitude matrix and an importance score matrix, wherein the weight values ​​in the weight magnitude matrix are normalized and the importance scores in the importance score matrix are normalized. Correlation analysis was performed on the weight magnitude matrix and importance score matrix to obtain the correlation coefficient; If the correlation coefficient reaches the preset strong correlation threshold, then sort the weights according to their magnitude, take the weights with the preset pruning ratio from the last position to the front, set the sparse mask of the corresponding weight position to 0, and set the sparse mask of other weight positions to 1. If the correlation coefficient is less than the preset strong correlation threshold, the comprehensive score S for each weight is calculated using the following formula: S = α × W norm +(1-α)×I norm Among them, W norm I represents the normalized magnitude of the weight. norm This represents the normalized importance score of the weight; α is a proportional parameter between 0 and 1, which is set according to the calculated value of the weight pruning risk. The greater the weight pruning risk, the closer the set value of α is to 0, and the smaller the weight pruning risk, the closer the set value of α is to 1. Based on the comprehensive score, the weights with a preset pruning ratio are taken from the last position to the front, and the sparse mask of the corresponding weight position is set to 0, while the sparse mask of other weight positions is set to 1.

8. The dynamic memory compression and layer scheduling method according to claim 7, characterized in that, The median line is used to divide the two-dimensional space of weight magnitude and importance score into four quadrants. The proportion of weights in the safe quadrant (where both weight magnitude and importance score are small) is determined, and the weight pruning risk in the safe quadrant is quantified. This includes calculating the risk factor Risk for each weight in the safe quadrant using the following formula: Risk = 1 - [(1 - W norm )×(1-I norm )], W norm I represents the normalized magnitude of the weight. norm The normalized importance score of the weight is used to calculate the average value of the risk factors of all weights within the safety quadrant as the weight pruning risk. Based on the weight percentage in the safety quadrant region, the pruning ratio is set to be less than the weight percentage in the safety quadrant region.

9. The dynamic memory compression and layer scheduling method according to claim 1, characterized in that, The large model is a Transformer model, and the subsequent model layers are determined in the following way: Define the current model layer as the nth model layer, predict at least the (n+1)th model layer and the (n+2)th model layer as subsequent model layers, and predict the access probability of the (n+1)th model layer as a preset first access probability value, and predict the access probability of the (n+2)th model layer as a preset second access probability value, wherein the first access probability value is greater than the second access probability value.

10. The dynamic memory compression and layer scheduling method according to claim 1, characterized in that, The large model is either a hybrid expert model or a conditional branching model, and the subsequent model layers are determined based on access history and contextual features. A pre-trained 2-layer MLP model outputs the probability distribution of each model layer based on the temporal features, content features, and load features corresponding to the current model layer. The temporal features include the historical path between model layers, the content features include the semantic and length information of the token, and the load features include system resources and task type information. The top few model layers, sorted by probability value, are used as the subsequent model layers for prediction. Other model layers, except for the predicted subsequent model layers, are stored using a preset high compression ratio. If the actual subsequent model layer is inconsistent with the predicted subsequent model layer, then the weight parameters of the MLP model are optimized.

11. The dynamic memory compression and layer scheduling method according to any one of claims 1 to 10, characterized in that, At the computation entry point, the hardware decompression unit performs real-time decompression, and the decompression operation is executed in parallel with the computation pipeline of the current model layer.

12. The dynamic memory compression and layer scheduling method according to any one of claims 1 to 10, characterized in that, In the process of dynamically swapping in and out of the model layer using a sliding window strategy, several resident model layers are also configured, and the on-chip main memory continuously stores the data of the resident model layers after asymmetric quantization compression.

13. A dynamic memory compression and layer scheduling system based on sparse awareness, characterized in that, Includes the following modules: The weight file module is configured to store the weight information of a trained large model, wherein the weights are defined as the strength values ​​of the neurons connected in the large model. The importance analysis module is configured to analyze the degree of influence of each weight on the accuracy of the model output, so as to divide the weights into key weights and secondary weights. An asymmetric quantization compression module is configured to perform asymmetric quantization compression on large model weights, wherein a first quantization precision is applied to the key weights and a second quantization precision is applied to the minor weights, and the first quantization precision is higher than the second quantization precision. A sparse mask generation module is configured to perform sparse feature analysis on the weights and generate a sparse mask based on the analyzed sparse features. The positions of weights with non-zero values ​​in the sparse mask are marked as corresponding valid weights, and the positions of weights with zero values ​​in the sparse mask are marked as corresponding invalid weights. The prediction and prefetching module is configured to predict the access probability of each model layer in order to determine the subsequent model layer whose access probability reaches the preset condition. During the calculation of the current model layer, the module performs the operation of prefetching the weight data of the subsequent model layer in parallel to obtain the data corresponding to the weight after asymmetric quantization compression. An off-chip storage module is configured to store the prefetched weight data of subsequent model layers into off-chip main memory; The bus transmission module is configured to transmit data stored in the off-chip main memory in a compressed state, and during the bus transmission process, read the data after asymmetric quantization compression of the effective weights from the off-chip main memory according to the sparse mask, or during the data storage process of the off-chip storage module, load the data after asymmetric quantization compression of the effective weights according to the sparse mask and store it in the off-chip main memory. The decompression module is configured to decompress the data transmitted on the bus at the calculation entry point using a hardware decompression unit to obtain the decompressed weight data. The dynamic layer scheduling module is configured to manage the dynamic swapping in and out of model layers using the following sliding window strategy: after the current model layer has been calculated, the weight data of the current model layer is removed from the on-chip main memory or marked as overwriteable, and the weight data of the subsequent model layer is extracted from the decompressed weight data and stored in the on-chip main memory. The data in the on-chip main memory is configured to be sent to the physical execution engine of the large model for inference.

14. The dynamic memory compression and layer scheduling system based on sparse awareness according to claim 13, characterized in that, It also includes a dynamic memory pool unit, which is configured to maintain the compression status, sparsity and access frequency of each data block at the memory block granularity, and dynamically adjust the compression level; And / or, the system supports the simultaneous deployment and scheduling of multiple large model instances on a single end device.

15. A portable AI device, characterized in that, Includes the dynamic memory compression and layer scheduling system as described in claim 13 or 14, or performs the dynamic memory compression and layer scheduling method as described in any one of claims 1 to 12.

Images