Neural network compression method for space computing equipment
By blocking, clustering and remapping the neural network weight matrix, the compatibility of neural network compression and storage architecture on spatial computing devices is solved, efficient model compression and energy efficiency improvement are achieved, and the application of edge computing scenarios is supported.
Patent Information
- Application Number
- CN202411947152.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-15
AI Technical Summary
The existing neural network compression methods are not compatible with the integrated storage and computing architecture, resulting in high latency when writing parameter data on spatial computing devices, affecting performance.
The weight matrix is clustered through the K-means algorithm, and the index matrix, Alpha matrix and Bias matrix are used for remapping and reconstruction, reducing data handling and improving energy efficiency.
It realizes large-scale weight compression of neural network models, improves hardware resource utilization and task throughput, adapts to the high-energy-efficiency characteristics of the integrated storage and computing architecture, and supports applications in edge computing scenarios.
Smart Images
Figure CN119830984A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of lightweight design of artificial intelligence algorithms, and in particular to a neural network compression method for spatial computing devices. Background Art
[0002] With the in-depth research on artificial intelligence algorithms and non-von Neumann hardware architectures, many new hardware and systems have emerged in academia and industry. They are represented by the rapidly iterative development of GPUs, multi-core chips, many-core chips, deep learning accelerators, neuromorphic chips, etc. Different from traditional general-purpose processors such as CPUs and GPUs that time-division multiplex computing resources in the form of instructions, spatial computing devices such as FPGAs, integrated storage and computing chips, and TPUs divide resources on a spatial scale and have the advantage of high energy efficiency. They have important application scenarios for edge computing scenarios (such as the Internet of Things, smart manufacturing, smart homes, smart cities, etc.).
[0003] In recent years, artificial intelligence technology has great potential for empowering smart grids, smart construction, and smart cities. This has brought huge benefits to edge computing scenarios. However, the scaling law drives artificial intelligence models toward larger parameters and higher computational loads, which greatly increases the requirements for computing performance and energy efficiency for computing devices. Highly flexible spatial computing devices have brought huge space for improving the parallel optimization of neural network tasks. However, this also brings serious challenges to the task mapping compression of neural networks.
[0004] Different from traditional time-division multiplexed parallel computing devices such as CPUs and GPUs, spatial computing devices have limited storage resources for a single device. Therefore, it is necessary to rely on lightweight methods of neural networks to achieve efficient model compression. One idea is to use feature clustering to compress model parameters, but existing compression methods are not compatible with spatial computing architectures represented by storage-computing integrated architectures. The reason is that although the spatial computing architecture has higher computing and reasoning performance, it has high data write latency, and frequent parameter data writes have a huge impact on performance, which cannot meet usage requirements. Summary of the invention
[0005] The purpose of the present invention is to provide a neural network compression method for spatial computing devices, which can effectively improve the utilization of hardware resources and improve task throughput, so that high-efficiency spatial computing devices can empower smart cities and smart construction, bringing new ideas to solve the problems raised by the above-mentioned background technology.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] A neural network compression method for a spatial computing device comprises the following steps:
[0008] S1: Matrix partitioning and normalization stage: Assume that the size of the original matrix is (I·K) 2 , the weight matrix is K 2 The size of the matrix is divided into blocks, and the sub-matrices are clustered and compressed based on the data distribution in the sub-matrices, including the relative and absolute values of the sub-matrices;
[0009] S2: Weighted clustering stage: Based on the K-means algorithm, for each matrix in the data set, the distance between each matrix and each cluster center is calculated and assigned to the cluster center with the nearest distance;
[0010] S3: Remapping stage: Considering the influence of the absolute value of each sub-matrix on the reconstruction accuracy, assuming that all values within a single weight are considered as Gaussian distribution, remapping maps the mean and variance of a sub-weight block distribution to the values before compression to compensate for the elimination of absolute value information in the normalization stage;
[0011] S4: Matrix reconstruction stage: It includes four parts based on index matrix, Alpha matrix, Bias matrix, and weight dictionary tensor. For each block B k , find the basis matrix Bc in the weight dictionary through the index matrix k , and then pass the basis matrix through Alpha k *Bc k +Bias k The reconstruction is performed in the same way as the inverse process of the compression transformation.
[0012] Furthermore, in S1, the lowest and highest values of each submatrix after block division are mapped to 0-255. The clustering object is the distribution of the relative values of the submatrices in the weights. The feature extraction of the absolute value depends on the remapping stage.
[0013] Furthermore, for a block with a size of K 2 The data block B, the maximum value in the data block is max(B), the minimum number is min(B), then the number of the normalized matrix Bnorm on the [i, j] coordinate is mapped to:
[0014] Bnorm[i,j]=(B[i,j]-min(B)) / max(B)*255.
[0015] Furthermore, in S2, the matrix Euclidean distance is used as the distance metric, and the calculation formula is as follows:
[0016]
[0017] Among them, c i is the i-th sub-matrix set S iThe cluster center matrix of i The mean of all matrices in , i belongs to [1, C]. After one round of iteration, for each cluster, the submatrix set S i , recalculate its cluster center, and the new cluster center is the mean of all data points in the cluster.
[0018] Furthermore, the specific calculation method in S3 is as follows: for a submatrix B of a block in the original matrix k The corresponding cluster center Bc k , assuming B k The data distribution is N(μ B ,σ B ), and the corresponding cluster center Bc k =N(μ Bc ,σ Bc ), and then use the basic transformation of normal distribution to transform Bc k Re-linearly map to B k In the corresponding distribution of , that is:
[0019] alpha k *Bc k +bias k ~B k
[0020] Corresponding alpha k and beta k The matrix is composed of linear transformation matrix Alpha matrix and Bias matrix, where k belongs to [1,I 2 ].
[0021] Compared with the prior art, the present invention has the following beneficial effects:
[0022] 1. The present invention realizes a large-scale weight compression technology for neural network models, especially large language models. The corresponding method is based on clustering of similar weight matrix features. There is no conflict with other existing technologies such as low-precision quantization, and the corresponding technical solutions can be superimposed and optimized.
[0023] 2. The present invention realizes the adaptive design of the spatial computing architecture with integrated storage and computing and pulsating array basic computing units, uses two-dimensional tensor blocks as compression granularity, reduces data movement in the spatial computing architecture, and realizes high energy efficiency ratio neural network reasoning.
[0024] 3. The present invention reserves corresponding clustering hyperparameters to ensure that the solution can be adjusted according to the trade-off between actual computing resource limitations and reasoning accuracy, and has a certain degree of design versatility. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1is a flow chart of the method of the present invention;
[0026] Figure 2 This is a diagram showing the effect of verifying the feasibility of the compression scheme of the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] like Figure 1 The compression method shown demonstrates how the single-layer weights of a neural network performing fully connected calculations are compressed under the method of the present invention and reconstructed during the calculation process. Figure 1 In the example, a two-dimensional tensor W∈R K×I×K×I For example, this method uses R K×K The weights are divided into I×I blocks as the basic block granularity. Afterwards, the I×I matrix blocks are clustered into C classes based on the data pattern, and the index of each block in this C class is established. In addition, two parameter matrices (Alpha matrix and Bias matrix) of size I×I need to be established to correct the mapping effect. In general, (3×I×I+C×K×K) / (K×K×I×I) can be achieved. For large language models represented by LLAMA 2, 85% storage compression can be achieved.
[0029] Based on this, an embodiment of the present invention provides a neural network compression method for spatial computing devices, including four stages: matrix partitioning and normalization, weight clustering, remapping, and matrix reconstruction. The entire process uses a single two-dimensional weight matrix as output. The output compressed matrix includes four parts: index matrix, Alpha matrix, Bias matrix, and weight dictionary three-dimensional tensor. The total storage occupancy after compression is much smaller than the initial matrix. Assume that the size of the original matrix is (I·K) 2 , in K 2 After partitioning and clustering the matrix for size, we try to use only C feature matrices (C is much smaller than I 2 ) is composed of a storage capacity of C·K 2 The weight dictionary is composed of the position index matrix (size I) corresponding to each block in the original matrix in the weight dictionary 2 ), and the linear transformation matrix Alpha matrix and Bias matrix (both of size I) for efficient reconstruction 2), realizes data compression of the original matrix, ensuring that it is friendly to the spatial architecture.
[0030] The specific steps are as follows:
[0031] S1: Matrix partitioning and normalization stage:
[0032] The weight matrix is K 2 After the size of the matrix is divided into blocks, the sub-matrices are clustered based on the data distribution in the sub-matrices to achieve the compression effect. The lowest and highest values of each sub-matrix after the block are mapped to 0-255 to reduce the influence of the absolute value of the value on the clustering. The clustering object is the distribution of the relative values of the sub-matrices in the weight rather than a single absolute value. The feature extraction of the absolute value depends on the remapping stage. For a certain size of K after the block 2 The data block B, the maximum value in the data block is max(B), the minimum number is min(B), then the number of the normalized matrix Bnorm on the [i, j] coordinate is mapped to:
[0033] Bnorm[i,j]=(B[i,j]-min(B)) / max(B)*255.
[0034] S2: Weighted clustering stage: Taking the K-means algorithm as an example, all matrices are clustered unsupervisedly. In the embodiment of the present invention, the number of clusters is an adjustable hyperparameter. In the K-means algorithm, for each matrix in the data set, its distance from each cluster center is calculated and assigned to the cluster center closest to it. This step usually uses the matrix Euclidean distance as the distance metric, and the calculation formula is as follows:
[0035]
[0036] where c i is the i-th sub-matrix set S i The cluster center matrix, whose value is equal to S i The mean of all matrices in , i belongs to [1, C]. After one round of iteration, for each cluster, the submatrix set S i , recalculate its cluster center, and the new cluster center is the mean of all data points in the cluster.
[0037] S3: Remapping stage: The absolute value of each submatrix is taken into account to ensure reconstruction accuracy while compressing. In this step, it is assumed that all values within a single weight are Gaussian distributed. The remapping aims to map the mean and variance of a sub-weight block distribution to the values before compression to compensate for the elimination of absolute numerical information in the normalization stage. For a submatrix B of a block in the original matrix k The corresponding cluster center Bck , assuming B k The data distribution is N(μ B ,σ B ), and the corresponding cluster center Bc k =N(μ Bc ,σ Bc ), and then use the basic transformation of normal distribution to transform Bc k Re-linearly map to B k In the corresponding distribution of , that is:
[0038] alpha k *Bc k +bias k ~B k
[0039] Corresponding alpha k and beta k The matrix formed is the Alpha matrix and Bias matrix mentioned above, where k belongs to [1,I 2 ].
[0040] S4: Reconstruction phase: The matrix reconstruction phase includes four parts based on the index matrix, Alpha matrix, Bias matrix, and weight dictionary tensor. For each block B k , find the basis matrix Bc in the weight dictionary through the index matrix k , before passing it through Alpha k *Bc k +Bias k The reconstruction is performed in the form of a compression transform, which is the inverse process of the above compression transform.
[0041] From the above, it can be seen that: the method in the embodiment of the present invention uses blocks as the compression granularity to meet the needs of soft and hard coordination; secondly, normalization ensures that the numerical pattern is used as the clustering standard, and the absolute value improves the clustering accuracy; in addition, the linear transformation of the distribution of data blocks based on the Gaussian distribution assumption is of great significance for improving compression efficiency.
[0042] The feasibility of the scheme depends on the existence of similarity patterns in the existing neural network weights. In order to further verify the feasibility of the method of the present invention, refer to Figure 2 As shown in the figure, for the model weights of LLAMA 2, with K being 128, 64, 32, and 16, the mean and standard deviation of the matrix after partitioning are statistically analyzed at each point. Figure 2 The visualization effect of the mean matrix is shown, and the darker the color, the larger the value (the standard deviation matrix shows similar characteristics and is not shown here).
[0043] The results show that for matrices with K=32 and above, significant repetitive patterns can be shown, which means that in large language models, Transformer weights have obvious repetitive patterns. Therefore, the corresponding repetitive patterns provide feasibility for the clustering method of the present invention to achieve a better compression ratio.
[0044] To sum up: The neural network compression method for spatial computing devices provided by the present invention fully considers the high energy-efficiency computing characteristics of the spatial computing architecture, and promotes the broad application prospects of artificial intelligence technology in edge computing scenarios such as smart cities, the Internet of Things, and intelligent manufacturing.
[0045] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A neural network compression method for spatial computing devices, characterized in that: The following steps are involved: S1: Matrix partitioning and normalization stage: Assume that the size of the original matrix is (I·K) 2 , the weight matrix is K 2 The size of the matrix is divided into blocks, and the sub-matrices are clustered and compressed based on the data distribution in the sub-matrices, including the relative and absolute values of the sub-matrices; S2: Weighted clustering stage: Based on the K-means algorithm, for each matrix in the data set, the distance between each matrix and each cluster center is calculated and assigned to the cluster center with the nearest distance; S3: Remapping stage: Considering the influence of the absolute value of each sub-matrix on the reconstruction accuracy, assuming that all values within a single weight are considered as Gaussian distribution, remapping maps the mean and variance of a sub-weight block distribution to the values before compression to compensate for the elimination of absolute value information in the normalization stage; S4: Matrix reconstruction stage: It includes four parts based on index matrix, Alpha matrix, Bias matrix, and weight dictionary tensor. For each block B k , find the basis matrix Bc in the weight dictionary through the index matrix k , and then pass the basis matrix through Alpha k *Bc k +Bias k The reconstruction is performed in the same way as the inverse process of the compression transformation.
2. A neural network compression method for spatial computing devices as claimed in claim 1, characterized in that: In S1, the lowest and highest values of each submatrix after block division are mapped to 0-255. The clustering object is the distribution of the relative values of the submatrices in the weights. The feature extraction of absolute values depends on the remapping stage.
3. A neural network compression method for spatial computing devices as claimed in claim 2, characterized in that: For a certain size K after block division 2 The data block B, the maximum value in the data block is max(B), the minimum number is min(B), then the number of the normalized matrix Bnorm on the [i, j] coordinate is mapped to: Bnorm[i,j]=(B[i,j]-min(B)) / max(B)*255.
4. A neural network compression method for spatial computing devices as claimed in claim 1, characterized in that: S2 uses matrix Euclidean distance as the distance metric, and the calculation formula is as follows: Among them, c i is the i-th sub-matrix set S i The cluster center matrix of i The mean of all matrices in , i belongs to [1, C]. After one round of iteration, for each cluster, the submatrix set S i , recalculate its cluster center, and the new cluster center is the mean of all data points in the cluster.
5. A neural network compression method for spatial computing devices as claimed in claim 1, characterized in that: The specific calculation method in S3 is as follows: For a submatrix B of a block in the original matrix k The corresponding cluster center Bc k , assuming B k The data distribution is N(μ B ,σ B ), and the corresponding cluster center Bc k =N(μ Bc ,σ Bc ), and then use the basic transformation of normal distribution to transform Bc k Re-linearly map to B k In the corresponding distribution of , that is: alpha k *Bc k +bias k ~B k Corresponding alpha k and beta k The matrix is composed of linear transformation matrix Alpha matrix and Bias matrix, where k belongs to [1,I 2 ].