Bit-width-aware weight compression method, decompression method, unit and device

Abstract

The application relates to the technical field of artificial intelligence, and provides a weight compression method, a decompression method, a unit and equipment with bit width perception, wherein the compression method comprises the following steps: dividing weight parameters of an artificial intelligence model into a plurality of weight groups, analyzing a numerical range in each weight group, and calculating minimum effective bit widths required for covering the numerical range; generating a corresponding non-zero value bitmap for each weight group; the non-zero value bitmap is used for recording the position distribution of non-zero weight values and zero values in the weight group; and the non-zero weight data in each weight group is arranged according to the corresponding minimum effective bit width, so as to form compressed weight data. The application can adaptively match the variable bit width demand caused by the change of repeated high-bit symbols in different weight parameters; by generating a non-zero value bitmap for each weight group, the all-zero numerical overhead in the weight matrix is accurately identified and directly removed while the bit width is dynamically compressed, so that the storage density is improved, and the external memory overhead and system energy consumption are reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to bit-width-aware weighted compression methods, decompression methods, units, and devices. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence (AI) technology, various neural network models have been widely applied in edge computing scenarios such as autonomous driving and embedded intelligent devices. In these edge intelligence scenarios, devices often face strict power consumption and bandwidth budget constraints.

[0003] As core data in AI accelerators, the storage size of weight parameters increases with the complexity of the model. During model inference, a large number of weight parameters need to be frequently read from external memory to the on-chip processing unit. This external memory access (EMA) operation consumes extremely high memory bandwidth and a large amount of system energy, severely restricting the energy efficiency and real-time performance of AI hardware accelerators, and becoming a key bottleneck for the large-scale deployment of general AI models on edge devices.

[0004] To reduce EMA overhead, several weight compression techniques exist, but all have significant limitations: 1. Weight compression scheme based on fixed bit width segmentation. This scheme typically presets a fixed bit width threshold and stores the weight parameters in two parts. However, in actual AI models, the weight parameters have a large number of repetitive high-order sign bits, and the effective bit widths of different parameters vary. The fixed bit width scheme cannot dynamically adapt to these differences and lacks the ability to optimize for zero-value redundancy, resulting in insufficient compression.

[0005] 2. Weight compression scheme based on non-zero bitmap. This scheme removes zero values ​​by recording all zero bits. However, this scheme ignores the compression space of non-zero values ​​themselves and cannot compress the large number of redundant high bits in non-zero values; especially in AI models with medium to low sparsity, its compression gain is very limited.

[0006] 3. Weight compression schemes for specific data distributions. This scheme optimizes only a single data distribution. However, the weights of general AI models often exhibit multiple complex distribution patterns in actual hardware deployments, which limits the applicability of this scheme in general scenarios and prevents it from providing sufficient and stable compression results. Summary of the Invention

[0007] This application provides a bit-width-aware weighted compression method, decompression method, unit, and device to address the technical problem that the EMA overhead of AI models in the prior art is too large, which cannot meet the requirements of high energy efficiency and low latency for edge deployment.

[0008] This application provides a bit-width-aware weight compression method, comprising: dividing the weight parameters of an artificial intelligence model into several weight sets; analyzing the numerical range within each weight set; and calculating the minimum effective bit width required to cover the numerical range; generating a corresponding non-zero value bitmap for each weight set; wherein the non-zero value bitmap is used to record the positional distribution of non-zero weight values ​​and zero values ​​in the weight set; and arranging the non-zero weight data in each weight set according to the corresponding minimum effective bit width to form compressed weight data.

[0009] According to the bit-width-aware weight compression method provided in this application, after arranging the non-zero weight data in each weight group according to the corresponding minimum effective bit width to form compressed weight data, the method further includes: storing the compressed address information separately from the compressed weight data; wherein, the compressed address information includes the minimum effective bit width and non-zero value bitmap of each weight group.

[0010] According to the bit-width-aware weight compression method provided in this application, the non-zero weight data in each weight set are arranged according to the corresponding minimum effective bit width, including: if the bit width exceeds the preset single row and column width of the memory when arranging a set of weights, then a new row of the memory is enabled to continue the arrangement.

[0011] According to the bit-width-aware weighted compression method provided in this application, the group size of the weight group is dynamically and flexibly configured according to the actual application scenario and hardware storage requirements in the artificial intelligence model, so as to achieve fine-grained dynamic adaptation of the bit width.

[0012] This application also provides a bit-width-aware weight decompression method, comprising: reading compression address information from memory, parsing the minimum effective bit width and non-zero value bitmap of each group of weights, and calculating the overall total bit width of the group of weights; dynamically updating the pointer according to the overall total bit width, and extracting the corresponding effective bit data from the compressed weight data; shifting and padding the effective bit data with zeros, and expanding it to the bit width required for the original calculation; parsing bit by bit according to the non-zero value bitmap, restoring the expanded effective bit data to the original weight position order, and outputting the decompressed weight data.

[0013] According to the bit-width-aware weight decompression method provided in this application, the pointer is dynamically updated according to the total bit width, and the corresponding valid bit data is extracted from the compressed weight data. The method includes: updating the start pointer and the end pointer according to the total bit width, extracting the corresponding valid bit data from the compressed weight data using the start pointer and the end pointer, and truncating redundant data.

[0014] According to the bit-width-aware weight decompression method provided in this application, the start pointer and end pointer are updated according to the total bit width, including: the start pointer is updated to the value of the end pointer when the previous set of weights ends, and the end pointer is updated to the value of the end pointer when the previous set of weights ends plus the total bit width.

[0015] This application also provides a bit-width-aware weight compression unit, comprising: an analysis module for dividing the weight parameters of an artificial intelligence model into several weight sets, analyzing the numerical range within each weight set, and calculating the minimum effective bit width required to cover the numerical range; a generation module for generating a corresponding non-zero value bitmap for each weight set; wherein the non-zero value bitmap is used to record the positional distribution of non-zero weight values ​​and zero values ​​in the weight set; and a compression module for arranging the non-zero weight data in each weight set according to the corresponding minimum effective bit width to form compressed weight data.

[0016] This application also provides a bit-width-aware weight decompression unit, comprising: a parsing module, used to read compressed address information from memory, parse the minimum effective bit width and non-zero value bitmap of each group of weights, and calculate the overall total bit width of the group of weights; an extraction module, used to dynamically update the pointer according to the overall total bit width, and extract the corresponding effective bit data from the compressed weight data; an expansion module, used to shift and pad the effective bit data with zeros, and expand it to the bit width required for the original calculation; and a decompression module, used to parse bit by bit according to the non-zero value bitmap, restore the expanded effective bit data to the original weight position order, and output the decompressed weight data.

[0017] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the bit-width-aware weight compression methods described above, or implements any of the bit-width-aware weight decompression methods described above.

[0018] This application provides a bit-width-aware weight compression method, decompression method, unit, and device. The compression method includes: dividing the weight parameters of an artificial intelligence model into several weight sets; analyzing the numerical range within each weight set; and calculating the minimum effective bit width required to cover the numerical range. A corresponding non-zero value bitmap is generated for each weight set; the non-zero value bitmap records the positional distribution of non-zero weight values ​​and zero values ​​in the weight set. The non-zero weight data in each weight set is arranged according to the corresponding minimum effective bit width to form compressed weight data. Through this method, this application can adaptively match the variable bit-width requirements caused by the repeated high-order signs of different weight parameters. By generating a non-zero value bitmap for each weight set, while dynamically compressing the bit width, it accurately identifies and directly eliminates the overhead of all-zero values ​​in the weight matrix, thereby improving storage density and reducing external memory access overhead and system power consumption. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the bit-width-aware weighted compression method provided in an embodiment of this application.

[0021] Figure 2 This is a flowchart illustrating the bit-width-aware weighted decompression method provided in an embodiment of this application.

[0022] Figure 3 This is a schematic diagram of the core steps of bit-width-aware compression and decompression provided in the embodiments of this application.

[0023] Figure 4 This is a flowchart illustrating the offline compression process provided in an embodiment of this application.

[0024] Figure 5 This is a schematic diagram of the bit-width-aware decompression unit design provided in an embodiment of this application.

[0025] Figure 6 This is a schematic diagram of the structure of the bit-width-aware weighted compression unit provided in an embodiment of this application.

[0026] Figure 7 This is a schematic diagram of the structure of the bit-width-aware weighted decompression unit provided in an embodiment of this application.

[0027] Figure 8 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0030] This application aims to address the technical problem of excessive External Memory Access (EMA) overhead caused by repeated access to weight parameters when deploying artificial intelligence models in hardware. Specifically, AI model weight parameters contain a large number of repeated high-order sign bits, and this bit width varies significantly across different parameter values. Existing weight compression and decompression methods lack effective handling for this variable bit width requirement, resulting in insufficient EMA compression. This, in turn, affects the energy efficiency and real-time performance of accelerators in edge intelligence scenarios (such as autonomous driving and embedded intelligent devices), limiting the deployment of general-purpose AI models on low-power devices.

[0031] This application proposes a bit-width-aware weight compression and decompression method, which can dynamically adapt to the variable bit-width requirements in the weight parameters of general AI models. With the help of on-chip real-time decompression circuit, it can significantly reduce the EMA requirement while ensuring the integrity of weight data, improve the energy efficiency and real-time performance of the accelerator, and adapt to the edge deployment requirements of AI models.

[0032] In this application, an artificial intelligence model refers to a computational model built based on deep learning or machine learning techniques for performing inference tasks. For example, an artificial intelligence model can be a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Transformer model, a Large Language Model (LLM), etc. An artificial intelligence model contains a large number of weight parameters, which are numerical values ​​learned during the model's training process and used to process the input data during the inference phase. It should be understood that an artificial intelligence model is not limited to the above examples and can be any machine learning or deep learning model that includes weight parameters.

[0033] In this application, bit width refers to the number of binary bits required to represent a numerical value. In deep learning models, weight parameters are typically stored with a fixed bit width (e.g., 8-bit, 16-bit, or 32-bit). However, the effective numerical range of many weight parameters is actually much smaller than the range that a fixed bit width can represent, leading to a significant waste of storage space. The bit width-aware method proposed in this application can solve this problem by dynamically determining the minimum effective bit width through analyzing the actual numerical range of the weight parameters, thereby achieving efficient weight compression.

[0034] This application provides a bit-width-aware weighted compression method. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating the bit-width-aware weighted compression method provided in an embodiment of this application. In this embodiment, the bit-width-aware weighted compression method may include steps S110 to S130, each step of which is detailed below: S110: Divide the weight parameters of the artificial intelligence model into several weight sets, analyze the numerical range within each weight set, and calculate the minimum effective bit width required to cover the numerical range.

[0035] Artificial intelligence models typically contain a large number of weight parameters; for example, a convolutional neural network may contain millions or even billions of weight parameters. To achieve fine-grained compression of these weight parameters, it is first necessary to divide them into several weight sets and analyze the numerical range of all weight values ​​within each weight set. The numerical range refers to the range corresponding to the largest absolute value in that weight set.

[0036] After determining the numerical range, calculate the minimum effective bit width required to cover that range. The minimum effective bit width is the minimum number of binary bits required to fully represent all weight values ​​within the group.

[0037] Optionally, the minimum effective bit width can be calculated by taking the logarithm of the maximum absolute value in the group to the base 2, rounding it up, and then adding one sign bit. However, the embodiments of this application are not limited to this, and other calculation methods that can determine the minimum representation bit width can also be used.

[0038] By calculating the minimum effective bit width, the weight values ​​within each weight set can be stored using just enough bits, instead of using a uniform fixed bit width, thus significantly reducing the storage space required.

[0039] S120: Generate a corresponding non-zero value bitmap for each weight reorganization; wherein, the non-zero value bitmap is used to record the positional distribution of non-zero weight values ​​and zero values ​​in the weight reorganization.

[0040] In the weight parameters of deep learning models, there are often a large number of zero values. A non-zero bitmap is a compact binary code used to mark whether the weight value at each position in the weight reassembly is zero.

[0041] For example, for the weighted combination [3,0,-1,0], its non-zero value bitmap is [1,0,1,0], where "1" indicates that the weight value at that position is non-zero, and "0" indicates that the weight value at that position is zero.

[0042] The purpose of the non-zero value bitmap is to: during compressed storage, only non-zero weight values ​​can be stored while zero values ​​are skipped, thereby further reducing storage space. Simultaneously, during decompression, the non-zero weight values ​​can be restored to their correct positions based on the non-zero value bitmap, and zero values ​​can be filled in at the zero value locations.

[0043] S130: Arrange the non-zero weight data in each weighted set according to the corresponding minimum effective bit width to form compressed weight data.

[0044] Specifically, after obtaining the minimum effective bit width and non-zero value bitmap for each weight group, the zero values ​​in the weight group can be removed, leaving only the non-zero weight values, and these non-zero weight values ​​can be arranged compactly according to the minimum effective bit width corresponding to the group.

[0045] In this way, the original fixed-width storage is replaced by a compact variable-width storage, different weight sets can use different effective bit widths, and zero values ​​no longer occupy storage space, thus achieving efficient weight compression.

[0046] According to the bit-width-aware weight compression method of this application embodiment, by dividing the weight parameters into several weight groups, analyzing the numerical range of each group and calculating the minimum effective bit width, generating a non-zero value bit map, and compactly arranging the non-zero weight data according to the minimum effective bit width, the storage space occupation can be greatly reduced while maintaining the accuracy of the weight data, and it is suitable for resource-constrained edge devices and embedded AI inference scenarios.

[0047] In some embodiments, the step of arranging the non-zero weight data in each weighted array according to the corresponding minimum effective bit width to form compressed weight data may further include: The compressed address information is stored separately from the compressed weight data; the compressed address information includes the minimum effective bit width and non-zero value bitmap of each weight group.

[0048] After weight compression is completed, this embodiment can store the metadata (i.e., compressed address information) used to guide decompression separately from the actual compressed weight data.

[0049] The compressed address information includes the minimum effective bit width of each weight group and the non-zero value bitmap of that group. The advantage of storing the compressed address information separately from the compressed weight data is that, during the decompression stage, the compressed address information can be read first to determine the effective bit width and non-zero value distribution of each weight group, and then the corresponding data can be accurately extracted from the compressed weight data based on this information.

[0050] For example, compressed address information can be stored in a dedicated area of ​​memory or a separate memory table, while the compressed weight data is stored contiguously in another area of ​​memory. This separate storage approach facilitates efficient pipelined processing in the hardware decompressor, allowing address resolution and data reading to be performed in parallel, thereby improving decompression throughput.

[0051] The embodiments of this application are not limited to this. The compressed address information and the compressed weight data can also adopt other storage organization methods, such as interleaved storage.

[0052] In some embodiments, the step of arranging the non-zero weight data in each weighted set according to the corresponding minimum effective bit width may specifically include: If the bit width exceeds the preset single-row column width of the memory when arranging a set of weights, a new row of memory is enabled to continue the arrangement.

[0053] In actual hardware memory, data is typically organized in rows, with each row having a fixed column width. When a set of non-zero weight values ​​is arranged according to the minimum effective bit width, the total bit width of the set of weights is equal to the number of non-zero weight values ​​multiplied by the minimum effective bit width of the set.

[0054] If the bit width does not exceed the single-row / column width of memory when arranging a set of weights, the weight data can be stored completely in one row of memory. However, if the bit width exceeds the single-row / column width of memory when arranging a set of weights, a new row of memory needs to be used to continue storing the remaining weight data.

[0055] This cross-line processing mechanism ensures that the compressed weighted data can be adapted to the physical structure constraints of the actual hardware memory, thereby guaranteeing the feasibility of hardware implementation.

[0056] Optionally, the column width of the memory can be set according to the specific hardware platform.

[0057] In some embodiments, the group size of the weight group is dynamically and flexibly configured according to the actual application scenario and hardware storage requirements in the artificial intelligence model, so as to achieve fine-grained dynamic adaptation of the bit width.

[0058] Specifically, different AI models, different network layers, and different hardware platforms may have different requirements for weight compression. For example, for network layers with relatively uniform weight distribution, a larger group size (e.g., 16 or 32 weight values ​​per group) can be used to reduce the overhead of compressing address information; for network layers with large differences in weight distribution, a smaller group size (e.g., 4 or 8 weight values ​​per group) can be used to achieve finer bit width adaptation, thereby obtaining a higher compression ratio.

[0059] For example, on embedded devices with extremely limited resources, a smaller packet size can be used to achieve a higher compression ratio in order to save limited storage space; while on platforms with more abundant storage resources, a larger packet size can be used to reduce the complexity of decompression and improve inference speed.

[0060] The group size can be set to, for example, 4, 8, 16, 32, etc., but the embodiments of this application are not limited to this. The group size can be set to any positive integer according to actual needs.

[0061] Furthermore, the group size can be configured differently between different layers of the same model, meaning that different layers can use different group sizes to achieve the best overall compression effect.

[0062] This application also provides a bit-width-aware weighted decompression method. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic flowchart of the bit-width-aware weighted decompression method provided in an embodiment of this application. In this embodiment, the bit-width-aware weighted decompression method may include steps S210 to S240, each step of which is as follows: S210: Read compressed address information from memory, parse the minimum effective bit width and non-zero bit map of each weight group, and calculate the overall total bit width of the weight group.

[0063] At the start of the decompression process, the compression address information is first read from memory. As mentioned earlier, the compression address information includes the minimum effective bit width and non-zero value bitmaps of each weight group. By parsing the compression address information, the minimum effective bit width and non-zero value bitmaps of the current weight group to be decompressed can be obtained.

[0064] After obtaining the minimum effective bit width and the non-zero value bitmap, the overall total bit width of the weight group can be calculated. The overall total bit width is equal to the number of non-zero weight values ​​in the group multiplied by the minimum effective bit width of the group. The number of non-zero weight values ​​can be obtained by counting the "1"s in the non-zero value bitmap.

[0065] The total bit width is used to guide the accurate extraction of the effective bit data of the group from the compressed weighted data.

[0066] S220: Dynamically update the pointer based on the total bit width and extract the corresponding valid bit data from the compressed weight data.

[0067] Since the compressed weight data consists of non-zero values ​​of each weight group arranged compactly according to their minimum effective bit width, pointers can be used to mark the position of the data to be read in the compressed weight data.

[0068] When reading each group of weighted data, the pointer is updated according to the total bit width of the group, thereby accurately locating and extracting the effective bit data of that group. Through this dynamic pointer update mechanism, the effective bit data of each group can be accurately extracted one by one from compactly arranged compressed data.

[0069] S230: Shift and pad the valid bits with zeros, and extend them to the bit width required for the original calculation.

[0070] Specifically, the valid bit data extracted in step S220 is compressed data stored according to the minimum valid bit width, which is usually smaller than the bit width required for the original calculation. In order for the decompressed weight data to participate in subsequent calculations, the valid bit data needs to be expanded to the bit width required for the original calculation.

[0071] Shifting and zero-padding refers to padding the high-order bits of the valid data with zeros, thus expanding its bit width to the original bit width. The expansion method can be, for example, zero-padding or sign-bit-padding based on the data representation (unsigned or signed number), but the embodiments in this application are not limited to these.

[0072] S240: Parse bit by bit according to the non-zero bitmap, restore the expanded valid bit data to the original weight position order, and output the decompressed weight data.

[0073] Specifically, the expanded valid bit data obtained in step S230 only contains non-zero weight values, while the zero values ​​in the original weight reassembly were removed during compression. Therefore, it is necessary to restore the non-zero weight values ​​to their correct positions in the original weight reassembly based on the non-zero value bitmap, and fill the zero value positions with zeros.

[0074] For example, for a weighted array with a non-zero bitmap of [1,0,1,0], the expanded effective bit data is [3,-1] (a total of 2 non-zero values). Then, the array is parsed bit by bit: the first bit is "1", outputting the first non-zero value 3; the second bit is "0", outputting the zero value 0; the third bit is "1", outputting the second non-zero value -1; the fourth bit is "0", outputting the zero value 0. The final decompressed weight data is [3,0,-1,0], consistent with the original weights.

[0075] In other embodiments, bit-by-bit parsing is not limited to sequential parsing over time, but can also be performed in parallel bit-by-bit parsing in some way; no specific limitation is made here.

[0076] Through the decompression process of steps S210 to S240 above, the compressed weight data can be accurately restored to the original weight data, thereby ensuring the correctness of the inference calculation.

[0077] In summary, this embodiment dynamically determines the minimum effective bit width by analyzing the actual numerical range of the weighted reassemblies and eliminates zero-value redundancy by combining a non-zero value bitmap, thus achieving efficient compression of the weight parameters of the artificial intelligence model. Compared with fixed bit width quantization methods in related technologies, this embodiment can achieve finer-grained bit width adaptation, significantly reducing storage overhead and data transmission bandwidth while maintaining model inference accuracy. It is particularly suitable for resource-constrained application scenarios such as edge computing and embedded AI inference.

[0078] Furthermore, the flexible configuration of group size enables the embodiments of this application to adapt to different model structures and hardware platforms, exhibiting good versatility and scalability.

[0079] In some embodiments, the step of dynamically updating the pointer according to the overall total bit width and extracting the corresponding valid bit data from the compressed weight data may specifically include: The start and end pointers are updated based on the total bit width. The start and end pointers are used to extract the corresponding valid bit data from the compressed weighted data and truncate redundant data.

[0080] To accurately extract the effective bit data for each weight group from the compactly arranged compressed data, this embodiment uses start and end pointers to mark the start and end positions of each data group in the compressed weight data. The start pointer points to the start position of the current group of effective bit data, and the end pointer points to the end position of the current group of effective bit data.

[0081] By using start and end pointers, the effective bits of the current group can be accurately extracted from the compressed weighted data, while redundant data that does not belong to the current group can be truncated.

[0082] In some embodiments, the step of updating the start pointer and end pointer according to the total bit width may specifically include: The start pointer is updated to the value of the end pointer when the previous set of weights ended, and the end pointer is updated to the value of the end pointer when the previous set of weights ended plus the total bit width.

[0083] Specifically, in the initial state of the decompression process, both the start pointer and the end pointer are initialized to 0. For the first group of weights, the start pointer remains at 0, and the end pointer is updated to 0 plus the total bit width of the first group. For each subsequent group of weights, the start pointer is updated to the value of the end pointer after the previous group of weights has been processed, that is, the start pointer of the current group is equal to the end pointer of the previous group; the end pointer is updated to the value of the end pointer when the previous group of weights has finished plus the total bit width of the current group.

[0084] By following this pattern, through this progressive pointer update mechanism, the effective bit data of each group can be accurately located and extracted in a continuous compressed data stream, achieving efficient streaming decompression.

[0085] The weight compression schemes in related technologies have the following drawbacks: 1. Fixed bit-width segmentation schemes have poor adaptability, cannot meet the different requirements of the actual effective bit width of AI model weight parameters, and do not have the ability to optimize zero-value redundancy. 2. The compression method for non-zero value bitmaps has the problem that its applicable scenarios are limited to a single optimization dimension. It can only remove zero values ​​for all zero points and cannot compress the redundant bit width of non-zero value cycles. Moreover, the compression gain is limited in medium to high sparsity AI models. 3. Compression methods for a specific type of data distribution have limited applicability. When weights are deployed on hardware, they may be split into multiple distribution modes, which cannot provide sufficient compression.

[0086] The bit-width-aware weight compression and decompression methods provided in this application can dynamically adapt to the variable bit-width requirements based on the repetition characteristics of high-order symbols in the weight parameters of the AI ​​model. By combining on-chip real-time decompression circuitry, the integrity of weight data is ensured while effectively reducing external memory access overhead, thereby improving the accelerator's energy efficiency and real-time processing capabilities, and better meeting the deployment requirements of AI models in edge computing scenarios.

[0087] The following is a complete numerical example illustrating the overall process of bit-width-aware weighted compression and decompression according to embodiments of this application. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of the core steps of bit-width-aware compression and decompression provided in the embodiments of this application.

[0088] The core of this embodiment is the bit-width-aware compression method and the supporting on-chip bit-width-aware decompression unit. By dynamically determining the minimum effective bit width, non-zero mapping record position, and compactly arranging storage through grouping, combined with on-chip real-time decompression, the EMA overhead of AI model weights is significantly reduced.

[0089] The overall solution includes two core components: offline compression and online decompression. It can be integrated into the AI ​​accelerator architecture, work in conjunction with existing processing units, and is compatible with various AI models such as CNN and Transformer.

[0090] Specifically, in the offline compression stage, the weight parameters of the general AI model are grouped and processed, the minimum effective bit width of each group of weights is dynamically calculated, zero-value redundancy is eliminated through non-zero mapping, and the data is stored in a compact arrangement according to bit width. That is, the offline stage of the fragment uses bit width-aware compression.

[0091] The interface enables off-chip access (EMA), during which compressed weights and address information can be obtained.

[0092] The weight data is restored in real time during the online decompression stage to ensure that the inference accuracy of the AI ​​model is not affected, while avoiding redundant EMA operations. In other words, a bit-width-aware decompression circuit is used in the on-chip online stage.

[0093] Please see Figure 4 , Figure 4 This is a flowchart illustrating the offline compression process provided in an embodiment of this application.

[0094] The bit-width-aware compression method is performed offline off-chip and consists of three steps.

[0095] First, weights are grouped to determine the minimum effective bit width for each group. Adjacent weight parameters are divided into several groups, and the group size can be flexibly adjusted. This can be configured in different AI models according to the actual application scenario and hardware storage requirements, laying the foundation for accurately calculating the minimum effective bit width.

[0096] Figure 4 The example uses groups of size 4 for illustration. For each weight parameter group, the original data may have high-order repetitions in the bit width, so it is necessary to analyze its numerical range and calculate the minimum bit width required to cover all weight values ​​in that group. Through such weight grouping, fine-grained dynamic adaptation of the bit width is achieved.

[0097] For example, the original data for Group0 is -24, -8, 0, 10, and the corresponding effective bit widths are 6, 4, 0, 5 respectively. That is, the weight values ​​of Group0 range from -24 to 10, which can be fully represented by 6 bits, and the minimum effective bit width of this group is 6.

[0098] The original data for the other group of values, Group1, is -3, -2, 0, 3. The corresponding effective bit widths are 3, 2, 0, 3, respectively. That is, the weight values ​​of Group1 range from -3 to 3, which can be represented by only 3 bits. Therefore, the minimum effective bit width of this group is 3.

[0099] Secondly, the positions of non-zero values ​​in each group are recorded. To maintain excellent compression performance even when the proportion of zero values ​​is high, this embodiment generates records of the non-zero value positions for each weight group, eliminating the need to store zero values ​​and further reducing redundant weights. Non-zero value positions are recorded using a bitmap, where 1 represents a non-zero value and 0 represents a zero value. This bitmap ensures that only non-zero weight data is retained during storage.

[0100] For example, Group0(-24,-8,0,10) generates a bitmap of 1101. Group1(-3,-2,0,3) generates a bitmap of 1101.

[0101] Finally, the weights are densely arranged. Each group of weights is arranged sequentially according to its defined minimum bit width, forming a compact storage format. If, during the arrangement, the bit width of a group of weights exceeds the single-row / column width of the memory, a new row is used to continue the arrangement, ensuring efficient utilization of storage resources. The single-row / column width of the memory can be flexibly configured according to the actual application requirements and the hardware support range.

[0102] During storage, the compressed address information (including minimum bit width information and non-zero value bitmap) of each group in the weight is stored separately from the compressed weight data to provide necessary information for subsequent decompression.

[0103] For example, for Group 0, the retained non-zero data are -24, -8, and 10. They are all formatted with the minimum significant bit width of 6 bits (6-bit) for this group and are arranged closely together. Next, the non-zero data of Group 1, -3, -2, and 3, are arranged. They are all formatted with the minimum significant bit width of 3 bits (3-bit) for this group and continue to be arranged after the data in Group 0.

[0104] In this way, data of different bit widths are seamlessly connected, maximizing the use of storage space. At the same time, the minimum bit width information of each group and the compressed address information such as non-zero value bitmaps are stored separately from these densely arranged weight data for subsequent decompression.

[0105] This embodiment achieves online real-time decompression through an on-chip bit-width-aware decompression unit. Please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a schematic diagram of the bit-width-aware decompression unit design provided in an embodiment of this application.

[0106] First, the compressed address information is read and the bit width is calculated. The decompression unit reads the compressed address information from the memory and parses out the minimum bit width information and non-zero value bitmap corresponding to each group of weights (1101 in the figure). The non-zero value bitmap is added bit by bit to obtain the number of 1s in the bitmap, which is also the number of non-zero values ​​(3 in the figure); then the result is multiplied by the minimum bit width of the group to obtain the overall total bit width of the weight group.

[0107] Secondly, pointer positioning and valid bit extraction are performed. The Begin and End pointer registers are dynamically updated using an adder. The Begin and End pointer registers record the start and end positions of the current group's weight value in memory, and are both initialized to 0.

[0108] Based on the calculated total bit width, the Begin and End pointer registers are updated: the Begin pointer register is updated to the current value stored in the End pointer register, representing the position where the previous weight group ended (i.e., the position where the current weight group began); while the End pointer register is updated to the current value stored in the End pointer register plus the total bit width, representing the position where the current weight group ended. The compressed weight row then precisely extracts the valid bit data (i.e., the "Expected Bits" in the figure: -3, -2, 3) based on the Begin and End pointer registers, and truncates unnecessary redundancy (i.e., the part marked "x" in the figure).

[0109] Next, shifting and bit width expansion are performed. The valid bits extracted in the previous step are moved to one end of the buffer through a shift operation, and the missing bit width at the other end is filled with zero values ​​using Padding 0s (the "pad" area in the diagram) to form the shifted weight row.

[0110] Since each value still retains the minimum bit width of the group during compression, the circuit will further perform a bit width expansion operation to expand it to the full bit width (BW-bit) to match the original hardware calculation format.

[0111] At this point, each data point still retains the minimum bit width of the group, so it can be expanded to the full bit width BW-bit (i.e., the bit width required for the original calculation) again through shift operations. This original bit width can be flexibly configured according to the model requirements and the calculation format supported by the hardware.

[0112] Finally, restore the weight positions. Based on the non-zero value bitmap, parse each bit to restore the non-zero values ​​to their original positions to ensure correct calculation. The specific steps for reading the non-zero value bitmap are as follows: If a "0" is encountered, it means that the position was originally zero, so a zero value is inserted (Insert 0), and the number of insertions is recorded (#insert). If a "1" is encountered, it indicates that the position is a non-zero valid value. The extended valid bit data is then shifted according to #insert-BW to restore the original weight position order. For example, the extended values ​​in the previous steps (such as -3, -2, 3) are shifted and filled in sequentially. Finally, the complete weight data is output for use by the processing unit (PE).

[0113] In summary, this embodiment utilizes a hardware-software co-optimization design to configure flexibly adaptable core parameters for the bit-width-aware compression method and decompression circuit. This achieves a dynamic balance between compression efficiency, decompression speed, and hardware resource consumption, adapting to the needs of different general AI models and edge deployment scenarios. Its core design parameters and co-operation logic are as follows: 1. GroupSize Group Size Parameter Design: As a core software configuration parameter in the offline compression process, GroupSize directly determines the granularity of weight grouping and needs to be adapted in conjunction with the hardware storage architecture and the weight distribution characteristics of the AI ​​model. When the difference in bit width of repeated high bits within a group of model weights is small, GroupSize can be set to a larger value to reduce the number of groups and redundancy in bit width information storage, thereby improving compression parallelism. When the bit width difference of model weights is large, GroupSize can be reduced to improve bit width adaptation accuracy and reduce invalid redundancy residue.

[0114] 2. Memory Bit Width Parameter Design: As a core hardware configuration parameter, the memory bit width is used for the compact arrangement of weights during off-chip offline compression and is related to the transmission bus bit width and the input interface bit width of the decompression module, forming a data transmission link of "storage-transmission-decompression". Based on model requirements and hardware limitations, different bus bit widths, on-chip buffer sizes, decompression module throughput, and memory bit widths will be designed collaboratively to ensure that the data consumption rate of each stage is consistent.

[0115] 3. Other collaborative parameter design: The storage bit width of the non-zero bitmap should correspond to the GroupSize to avoid redundant bit occupation; the pointer step size parameter in the decompression stage is linked with the memory bit width and GroupSize, and is dynamically adjusted by the top-level controller according to the minimum bit width information read in real time, to ensure that the pointer accurately positions the start and end positions of each group of compression weights, and realizes the timing synchronization of decompression and PE unit operation.

[0116] In summary, this embodiment employs a dynamic minimum bit width determination mechanism, which can accurately adapt to the bit width differences of each weight group, thereby improving the targeting of compression, significantly reducing the overhead of external memory access for weights, and ultimately achieving higher energy efficiency and lower latency.

[0117] Furthermore, by introducing a variable bit-width grouping mechanism, high-bit redundancy is further eliminated, which further reduces the overhead of external memory access for weights, resulting in better performance in terms of energy efficiency and latency.

[0118] This embodiment also combines a dual mechanism of "grouped variable bit width" and "non-zero value bitmap," which not only dynamically adapts to variable bit width requirements but also effectively eliminates zero-value weights, resulting in stronger data distribution adaptability. This design significantly reduces the overhead of external memory access for weights, thus offering clear advantages in improving energy efficiency and reducing latency.

[0119] Furthermore, this embodiment has been successfully designed, synthesized, and placed and routed using a standard 28nm process, and has undergone tape-out verification and testing. The performance indicators of this embodiment are shown below:

[0120] With the solution adopted in this embodiment, the chip's peak energy efficiency is 20.92 TOPS / W, the on-chip energy efficiency under a specific dataset is 3.87 TOPS / W, and the system energy efficiency considering EMA costs is 1.99 TOPS / W. Compared with the baseline results without using the solution of this embodiment, the EMA is reduced by 65%, energy consumption is reduced by 46%, and latency is reduced by 14% under the specific dataset.

[0121] This application also provides a bit-width-aware weighted compression unit. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the structure of the bit-width-aware weight compression unit provided in an embodiment of this application. In this embodiment, the bit-width-aware weight compression unit may include an analysis module 610, a generation module 620, and a compression module 630.

[0122] Analysis module 610 is used to divide the weight parameters of the artificial intelligence model into several weight sets, analyze the numerical range within each weight set, and calculate the minimum effective bit width required to cover the numerical range.

[0123] The generation module 620 is used to generate a corresponding non-zero value bitmap for each weighting reorganization; wherein, the non-zero value bitmap is used to record the position distribution of non-zero weight values ​​and zero values ​​in the weighting reorganization.

[0124] Compression module 630 is used to arrange the non-zero weight data in each weight set according to the corresponding minimum effective bit width to form compressed weight data.

[0125] In some embodiments, the bit-width-aware weighted compression unit may further include a storage module, which is specifically used for: The compressed address information is stored separately from the compressed weight data; the compressed address information includes the minimum effective bit width and non-zero value bitmap of each weight group.

[0126] In some embodiments, the compression module 630 may specifically be used for: If the bit width exceeds the preset single-row column width of the memory when arranging a set of weights, a new row of memory is enabled to continue the arrangement.

[0127] In some embodiments, the group size of the weight group is dynamically and flexibly configured according to the actual application scenario and hardware storage requirements in the artificial intelligence model, so as to achieve fine-grained dynamic adaptation of the bit width.

[0128] This application also provides a bit-width-aware weighted decompression unit; please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of the structure of the bit-width-aware weighted decompression unit provided in an embodiment of this application. In this embodiment, the bit-width-aware weighted decompression unit may include a parsing module 710, an extraction module 720, an expansion module 730, and a decompression module 740.

[0129] The parsing module 710 is used to read compressed address information from memory, parse the minimum effective bit width and non-zero value bitmap of each weight group, and calculate the overall total bit width of the weight group.

[0130] The extraction module 720 is used to dynamically update the pointer according to the overall total bit width and extract the corresponding valid bit data from the compressed weight data.

[0131] The extension module 730 is used to shift and pad the valid bits of data with zeros, and extend the bit width required for the original calculation.

[0132] The decompression module 740 is used to parse the non-zero bitmap bit by bit, restore the expanded valid bit data to the original weight position order, and output the decompressed weight data.

[0133] In some embodiments, the extraction module 720 is specifically used for: The start and end pointers are updated based on the total bit width. The start and end pointers are used to extract the corresponding valid bit data from the compressed weighted data and truncate redundant data.

[0134] In some embodiments, the extraction module 720 is specifically used for: The start pointer is updated to the value of the end pointer when the previous set of weights ended, and the end pointer is updated to the value of the end pointer when the previous set of weights ended plus the total bit width.

[0135] On the other hand, this application also provides an electronic device, please refer to... Figure 8 , Figure 8 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 8 As shown, the electronic device may include a memory 820, a processor 810, and a computer program stored in the memory 820 and executable on the processor 810. When the processor 810 executes the program, it can implement the above-described method, which may include: The weight parameters of the artificial intelligence model are divided into several weight sets. The numerical range within each weight set is analyzed, and the minimum effective bit width required to cover the numerical range is calculated. A corresponding non-zero value bitmap is generated for each weight set. The non-zero value bitmap is used to record the positional distribution of non-zero weight values ​​and zero values ​​in the weight set. The non-zero weight data in each weight set are arranged according to the corresponding minimum effective bit width to form compressed weight data. Alternatively, read the compressed address information from memory, parse the minimum effective bit width and non-zero value bitmap of each weight group, and calculate the overall total bit width of the weight group; dynamically update the pointer according to the overall total bit width, and extract the corresponding effective bit data from the compressed weight data; shift and pad the effective bit data with zeros, and expand it to the bit width required for the original calculation; parse bit by bit according to the non-zero value bitmap, restore the expanded effective bit data to the original weight position order, and output the decompressed weight data.

[0136] Optionally, the electronic device may further include a communication bus 830 and a communication interface 840, wherein the processor 810, the communication interface 840, and the memory 820 communicate with each other through the communication bus 830. The processor 810 can call the computer program in the memory 820 to execute the methods provided by the above methods.

[0137] Furthermore, the logical instructions in the aforementioned memory 820 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A bit-width-aware weighted compression method, characterized in that, include: The weight parameters of the artificial intelligence model are divided into several weight sets, the numerical range within each weight set is analyzed, and the minimum effective bit width required to cover the numerical range is calculated. A corresponding non-zero value bitmap is generated for each weighted reorganization; wherein, the non-zero value bitmap is used to record the positional distribution of non-zero weight values ​​and zero values ​​in the weighted reorganization; The non-zero weight data in each weighted set are arranged according to the corresponding minimum effective bit width to form compressed weight data.

2. The bit-width-aware weighted compression method according to claim 1, characterized in that, After arranging the non-zero weight data in each weighted group according to the corresponding minimum effective bit width to form compressed weight data, the method further includes: The compressed address information is stored separately from the compressed weight data; wherein, the compressed address information includes the minimum effective bit width and non-zero value bitmap of each weight group.

3. The bit-width-aware weighted compression method according to claim 1, characterized in that, The step of arranging the non-zero weight data in each weighted group according to the corresponding minimum effective bit width includes: If the bit width exceeds the preset single-row column width of the memory when arranging a set of weights, a new row of memory is enabled to continue the arrangement.

4. The bit-width-aware weighted compression method according to claim 1, characterized in that, The group size of the weight group is dynamically and flexibly configured according to the actual application scenario and hardware storage requirements in the artificial intelligence model, so as to achieve fine-grained dynamic adaptation of the bit width.

5. A bit-width-aware weighted decompression method, characterized in that, include: Read compressed address information from memory, parse the minimum effective bit width and non-zero value bitmap of each weight group, and calculate the overall total bit width of the weight group. The pointer is dynamically updated based on the overall total bit width, and the corresponding valid bit data is extracted from the compressed weight data. The valid bits are shifted and padded with zeros, and then expanded to the bit width required for the original calculation. The non-zero bitmap is parsed bit by bit to restore the expanded valid bit data to the original weight position order, and the decompressed weight data is output.

6. The bit-width-aware weighted decompression method according to claim 5, characterized in that, The step of dynamically updating the pointer based on the overall total bit width and extracting the corresponding valid bit data from the compressed weighted data includes: The start and end pointers are updated based on the total bit width. The start and end pointers are used to extract the corresponding valid bit data from the compressed weighted data and truncate redundant data.

7. The bit-width-aware weighted decompression method according to claim 5, characterized in that, The step of updating the start and end pointers based on the total bit width includes: The start pointer is updated to the value of the end pointer when the previous set of weights ends, and the end pointer is updated to the value of the end pointer when the previous set of weights ends plus the total bit width.

8. A bit-width-aware weighted compression unit, characterized in that, include: The analysis module is used to divide the weight parameters of the artificial intelligence model into several weight sets, analyze the numerical range within each weight set, and calculate the minimum effective bit width required to cover the numerical range. A generation module is used to generate a corresponding non-zero value bitmap for each weighting reorganization; wherein, the non-zero value bitmap is used to record the positional distribution of non-zero weight values ​​and zero values ​​in the weighting reorganization; The compression module is used to arrange the non-zero weight data in each weighted set according to the corresponding minimum effective bit width to form compressed weight data.

9. A bit-width-aware weighted decompression unit, characterized in that, include: The parsing module is used to read compressed address information from memory, parse the minimum effective bit width and non-zero value bitmap of each weight group, and calculate the overall total bit width of the weight group. The extraction module is used to dynamically update the pointer according to the overall total bit width and extract the corresponding valid bit data from the compressed weight data; An extension module is used to shift and pad the effective bit data with zeros, and extend it to the bit width required for the original calculation; The decompression module is used to parse the non-zero bitmap bit by bit, restore the expanded valid bit data to the original weight position order, and output the decompressed weight data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the bit-width-aware weight compression method as described in any one of claims 1 to 4, or implements the bit-width-aware weight decompression method as described in any one of claims 5 to 7.

Images