Data deployment system, training method, reasoning method and chip of deep neural network

Abstract

The invention relates to an intelligent integrated circuit and an intelligent chip in an artificial intelligence hardware platform. The invention relates to the field of deep neural networks in an artificial intelligence general technology, in particular to a data deployment system, a training method, an inference method and a chip of a deep neural network. The deep neural network data deployment system comprises a storage module; the matrix decomposition module is used for decomposing the original matrix into the sum of N sub-matrixes, and N is greater than or equal to 2; wherein one of the sub-matrices is further decomposed into a product of two or more low-rank matrices; the matrix expression processing module is used for expressing the sub-matrixes which are not decomposed at first precision and then storing the sub-matrixes; expressing the two low-rank matrixes with second precision and then storing the two low-rank matrixes; wherein the original matrix is a weight matrix and / or an activation value matrix, and the first precision is lower than the second precision. According to the method, on the premise that the performance of the model is basically not lost, the storage space required in the training and reasoning process is effectively reduced, and the calculation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to intelligent integrated circuits, intelligent chips, and AI chips in artificial intelligence hardware platforms; deep neural networks, multi-layer neural networks, convolutional neural networks and other fields in general artificial intelligence technologies, and in particular to a data deployment system, training method, reasoning method, and chip for a deep neural network. Background Art

[0002] Deep neural network (DNN) is an advanced technology in the field of machine learning. It is a neural network with at least one hidden layer. It can process and identify complex data by simulating the connection and information processing methods of human brain neurons. Compared with shallow networks, the advantage of deep neural networks is that their multi-layer structure can extract and combine high-level features in the original data, thereby automatically learning more effective representations, improving the performance and generalization ability of the model, and reducing the risk of overfitting. This multi-level feature extraction and combination capability makes deep neural networks more powerful in expression and has a strong advantage in dealing with highly complex problems.

[0003] However, the success of deep neural networks comes at the cost of high computational complexity. Since deep neural networks have a large number of parameters and complex computational processes, the huge number of parameters makes it difficult to read and store data, making training and reasoning more difficult, requiring more data and computing resources, and resulting in higher energy consumption. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] The present invention is expected to at least partially solve one of the above technical problems.

[0006] 2. Technical Solution

[0007] The first aspect of the present invention provides a deep neural network data deployment system. The deep neural network data deployment system of the present invention comprises:

[0008] Storage module;

[0009] A matrix decomposition module, used for decomposing an original matrix into a sum of N sub-matrices, N≥2; wherein at least one of the N sub-matrices is further decomposed into a product of two or more low-rank matrices, and the rank of the low-rank matrix is ​​less than the rank of the original matrix;

[0010] Matrix expression processing module, used for:

[0011] For the undecomposed submatrix, express it with the first precision and store it in the storage module;

[0012] For the decomposed sub-matrix, the corresponding low-rank matrix is ​​expressed with the second precision and stored in the storage module;

[0013] The original matrix is ​​a weight matrix and / or an activation value matrix of a deep neural network, and the first precision is lower than the second precision.

[0014] In some embodiments of the present invention, the storage module includes: a low-frequency read-write area and a high-frequency read-write area; wherein the undecomposed sub-matrix is ​​stored in the low-frequency read-write area of ​​the storage module, and the low-rank matrix corresponding to the decomposed sub-matrix is ​​stored in the high-frequency read-write area of ​​the storage module.

[0015] In some embodiments of the present invention, the storage module includes: an off-chip storage unit and an on-chip storage unit; wherein the undecomposed sub-matrix is ​​stored in the off-chip storage unit, and the low-rank matrix corresponding to the decomposed sub-matrix is ​​stored in the on-chip storage unit.

[0016] In some embodiments of the present invention, the second precision is: the original precision of the original matrix.

[0017] In some embodiments of the present invention, the low-rank matrix corresponding to the decomposed sub-matrix is ​​updated at a higher frequency than the undecomposed sub-matrix.

[0018] In some embodiments of the present invention, the original matrix is ​​a weight matrix and / or an activation value matrix and / or a key-value cache matrix obtained during the training process of a deep neural network in one of the following fields: speech recognition, natural language processing, text generation, dialogue systems, translation, document summarization, image generation, video generation, image recognition, and image segmentation.

[0019] In some embodiments of the present invention, it also includes: a matrix restoration module, which is used to restore the original matrix expressed with original precision based on the undecomposed submatrix expressed with the first precision and the low-rank matrix of the decomposed submatrix expressed with the second precision.

[0020] In some embodiments of the present invention, N=2; a matrix decomposition module is used to perform low-rank adaptive decomposition on the original matrix W to obtain: W=J+L×R; wherein: the sub-matrix that has not been decomposed is: the main feature matrix J, whose dimension is the same as the original matrix; the two low-rank matrices corresponding to the decomposed sub-matrices are: the second-first low-rank matrix L, the second-second low-rank matrix R, and the rank k of the two low-rank matrices is less than the rank of the original matrix; a matrix expression processing module is used to: express the main feature matrix J with the first precision and then store it; express the second-first low-rank matrix L and the second-second low-rank matrix R with the second precision and then store them.

[0021] In some embodiments of the present invention, in the matrix decomposition module, the rank k of the low-rank adaptive decomposition satisfies: 1≤k≤20%D, where D is the rank of the original matrix.

[0022] In some embodiments of the present invention, the rank k of the low rank adaptive decomposition is selected from one of the following values: 2, 3, 4, 8, 16, 32.

[0023] In some embodiments of the present invention, in the matrix decomposition module, the low-rank adaptive decomposition method is one of the following decomposition methods: singular value decomposition and eigenvalue decomposition.

[0024] In some embodiments of the present invention, in the matrix expression processing module, the main feature matrix J is stored with integer data precision.

[0025] In some embodiments of the present invention, in the matrix expression processing module, the second-first low-rank matrix L and the second-second low-rank matrix R are both expressed using one of the following data precisions and stored: Float32 type, Float64 type, Float16 type.

[0026] In some embodiments of the present invention, the storage module includes: a low-frequency read-write area and a high-frequency read-write area; a matrix expression processing module: stores the main feature matrix in the low-frequency read-write area of ​​the storage module; stores the second-first low-rank matrix and the second-second low-rank matrix in the high-frequency read-write area of ​​the storage module.

[0027] In some embodiments of the present invention, the low-rank adaptive decomposition method is: singular value decomposition; a matrix decomposition module is used to:

[0028] Perform singular value decomposition on the original matrix: W = UΣV T , where Σ is a diagonal matrix, the elements on its diagonal are singular values, and these singular values ​​are arranged in order of size;

[0029] Decompose Σ into the sum of two diagonal matrices Σ=Σ1+Σ2, where the first diagonal matrix Σ1 contains the first k singular values ​​and the second diagonal matrix Σ2 contains the remaining singular values;

[0030] Using the decomposed Σ, we can get: W = UΣV T =U(Σ1+Σ2)V T =UΣ1V T +UΣ2V T ;in:

[0031] UΣ1V T As the main feature matrix J;

[0032] U and Σ2V T As the second low-rank matrix L, the second low-rank matrix R respectively; or, UΣ2 and V T As the second-first low-rank matrix L and the second-second low-rank matrix R respectively;

[0033] Among them, the sum of squares of the singular values ​​in the first diagonal matrix Σ1 accounts for more than 95% of the sum of squares of the singular values ​​in the diagonal matrix Σ.

[0034] In some embodiments of the present invention, the low-rank adaptive decomposition method is: eigenvalue decomposition; a matrix decomposition module is used to:

[0035] Solve the eigenvalue diagonal matrix D of the original matrix and the corresponding eigenvector V, where D is a diagonal matrix and is arranged in descending order;

[0036] Decompose the original matrix into: W = VDV T ;

[0037] Decompose the diagonal matrix D: D = D1 + D2, and then W = V(D1 + D2)V T =VD1V T +VD2V T , where D1 retains the first k elements in the diagonal matrix D, and D2 retains the other elements in the diagonal matrix, where:

[0038] VD1V T As the main feature matrix J;

[0039] V and D2V T As the second low-rank matrix L and the second low-rank matrix R respectively; or, VD2 and V T They are respectively used as the second-first low-rank matrix L and the second-second low-rank matrix R.

[0040] In some embodiments of the present invention, N=3; the first precision includes: first-first precision; first-second precision; wherein, both the first-first precision and the second precision are lower than the second precision, and the first-first precision is lower than the first-second precision; a matrix decomposition module, used to perform low-rank adaptive decomposition on the original matrix W, and obtain: W=J1+J2+L×R; wherein, the low-rank adaptive decomposition is: singular value decomposition or eigenvalue decomposition: the undecomposed sub-matrices are: the first main characteristic matrix J1 and the second main characteristic matrix J2, whose dimensions are the same as the original matrix; the two low-rank matrices corresponding to the decomposed sub-matrices are: the second-first low-rank matrix L, the second-second low-rank matrix R, and the rank k of the two low-rank matrices is less than the rank of the original matrix; a matrix expression processing module, used to: express the first main characteristic matrix J1 with the first-first precision and store it; express the second main characteristic matrix J2 with the first-second precision and store it; express the second-first low-rank matrix L and the second-second low-rank matrix R with the second precision and store them.

[0041] In some embodiments of the present invention, N=2; a matrix decomposition module is used to perform a principal re-decomposition on the original matrix W to obtain: W=K+S×T; wherein: the sub-matrix that has not been decomposed is: the residual matrix K, whose dimension is the same as the original matrix; the two low-rank matrices corresponding to the decomposed sub-matrices are: the first low-rank matrix S, the first second low-rank matrix T, and the ranks of the two low-rank matrices are smaller than the rank of the original matrix; a matrix expression processing module is used to: compress and / or quantize the residual matrix K, and store it after expressing it with the first precision; and store the first low-rank matrix S and the first second low-rank matrix T after expressing them with the second precision.

[0042] In some embodiments of the present invention, in the matrix expression processing module, both the first low-rank matrix S and the first second low-rank matrix T are stored using Float32 or Float64 data precision.

[0043] In some embodiments of the present invention, in the matrix expression processing module, “expressing the residual matrix K with the first precision and then storing it” adopts at least one of the following methods:

[0044] The residual matrix K is expressed with integer data precision and then stored;

[0045] The residual matrix K is compressed using a sparse matrix compression method and then stored. The sparse matrix compression method is: lossless compression or lossy compression.

[0046] In some embodiments of the present invention, the storage module includes: an on-chip storage unit and an off-chip storage unit: a matrix expression processing module: storing the first low-rank matrix S and the first second low-rank matrix T in the on-chip storage unit; expressing the residual matrix K with the first precision and storing it in the off-chip storage unit.

[0047] In some embodiments of the present invention, the matrix decomposition module is used to: determine the rank k of the low-rank matrix, k≥1; determine the quantization bit width n of the weight, n≥1; perform singular value decomposition on the original matrix; W=UΣV T ; Where Σ is a diagonal matrix, the elements on the diagonal are singular values, and these singular values ​​are arranged in order of size; decompose Σ into the sum of two diagonal matrices Σ=Σ1+Σ2, where the first diagonal matrix Σ1 contains the first k singular values, and the second diagonal matrix Σ2 contains the remaining singular values; using the decomposed Σ, we get: W=UΣV T =U(Σ1+Σ2)V T =UΣ1V T +UΣ2V T ; Among them, UΣ2V T As the residual matrix K; U and Σ1V T As the first low-rank matrix S and the first second low-rank matrix T respectively; or, UΣ1 and VT As the first low-rank matrix S and the second low-rank matrix T respectively; take the maximum and minimum values ​​of the residual matrix K, and divide the value of the residual matrix K into 2 according to the quantization bit width n of the weight n Class, respectively expressed with integers, to obtain a new residual matrix, which is compressed using a sparse compression algorithm.

[0048] In some embodiments of the present invention, the rank k of the low-rank matrix satisfies: Wherein, H is the rank of the original matrix; the quantization bit width n of the weight satisfies: 2≤n≤8.

[0049] The second aspect of the present invention provides a deep neural network training method. The deep neural network training method includes: after the first iteration is completed, the obtained result is used as the original matrix, and the main feature matrix J, the second low-rank matrix L, and the second low-rank matrix R are stored by the above deep neural network data deployment system; after each iteration in the subsequent 2nd, ..., M-1 iterations is completed, the obtained result is used as the original matrix, the main feature matrix J is kept unchanged, and the matrix decomposition method in the deep neural network data deployment system is used: W = J + L × R, and two new low-rank matrices - the second low-rank matrix L, the second low-rank matrix R are obtained, and the second low-rank matrix L and the second low-rank matrix R are updated under the second precision expression, M≥3; after the Mth iteration is completed, the obtained result is used as the original matrix, and the above deep neural network data deployment system is used to obtain a new main feature matrix J, two low-rank matrices - the second low-rank matrix L, the second low-rank matrix R; under the first precision expression, the main feature matrix J is updated, and under the second precision expression, the second low-rank matrix L and the second low-rank matrix R are updated.

[0050] The third aspect of the present invention provides a deep neural network reasoning method. The deep neural network reasoning method includes: before reasoning, the weight matrix is ​​used as the original matrix, and the deep neural network data deployment system as above is used to store the residual matrix, the first low-rank matrix, and the first second low-rank matrix; during the reasoning process, the matrix to be calculated is received; the residual matrix expressed with the first precision is decompressed and / or dequantized to restore it to the residual matrix expressed with the original precision; the first low-rank matrix S and the first second low-rank matrix T are multiplied, and the multiplication result is added to the residual matrix expressed with the original precision to obtain the restored original matrix; the restored original matrix and the matrix to be calculated are used for reasoning calculation.

[0051] A fourth aspect of the present invention provides a deep neural network chip.

[0052] In some embodiments of the present invention, the deep neural network chip is configured to implement: the deep neural network data deployment system as above.

[0053] In some embodiments of the present invention, the deep neural network chip is configured to implement: the deep neural network training method as above;

[0054] In some embodiments of the present invention, the deep neural network chip is configured to implement: the deep neural network reasoning method as above.

[0055] 3. Beneficial Effects

[0056] It can be seen from the above technical solution that the present invention has at least one of the following beneficial effects compared with the prior art:

[0057] 1. The present invention proposes to use different precisions, different processing methods, and different locations for data with different amounts of information. Under the premise of maintaining the basic performance of the model, the storage space required in the training and reasoning process is effectively reduced, and the computing efficiency is improved, thereby achieving the goal of saving storage resources and speeding up the reasoning speed. In addition, the compressed model has lower energy consumption and is easier to deploy, which greatly improves the practicality and scalability of the model.

[0058] 2. Different from the traditional matrix decomposition method, the deployment method of the present invention takes into account the functional, price, and bandwidth constraints of different types of storage, so that the deployment of large language models can have high performance, high cost performance, and small area.

[0059] Specifically, in one embodiment of the present invention, through the low-rank adaptive decomposition of the weight (activation value matrix, key value cache) matrix, the main weight matrix and the low-rank matrix are deployed with different precisions and different storage locations. For the low-rank matrix that needs to be updated, high-precision (Float16, Float32 or Float64) memory is used for reading and writing to improve the computing efficiency; for the relatively fixed main feature matrix, a low-frequency read-write type of storage is used, and a longer time interval is used for updating.

[0060] 3. Different from the existing method that places the main information of the original matrix in the residual matrix here, the present invention places the main information of the matrix in a low-rank matrix to retain accuracy. Although the residual matrix is ​​relatively large, it adopts low-bit encoding, sparse matrix compression and other methods to greatly reduce the memory space occupied by the overall network, achieving a cost-effective design.

[0061] Specifically, in one embodiment of the present invention, through the main weight decomposition of the weight (activation value matrix, key value cache) matrix, the main matrix containing the main information and the residual matrix containing the secondary information are deployed with different precisions and different storage locations. The advantage of this is that the main components are saved with higher precision data during deployment, and the residual values ​​with less impact are compressed using a sparse matrix compression method to compress the components of the matrix in a low-bit manner to save it, thereby reducing the inference time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the structure of a deep neural network data deployment system according to an embodiment of the present invention.

[0063] Figure 2 for Figure 1 A schematic diagram showing the characteristics and corresponding storage methods of the main feature matrix and low-rank matrix in the low-rank adaptive decomposition in the deep neural network data deployment system.

[0064] Figure 3 This is a schematic diagram of the structure of the second embodiment of the deep neural network data deployment system of the present invention.

[0065] Figure 4 for Figure 3 The figure shows a schematic diagram of the characteristics and corresponding storage methods of the principal feature matrix and the low-rank matrix in the principal weight decomposition in the deep neural network data deployment system.

[0066] Figure 5 This is a schematic diagram of the respective characteristics and corresponding storage methods of the main feature matrix and the low-rank matrix in the low-rank adaptive decomposition in the third embodiment of the deep neural network data deployment system of the present invention.

[0067] Figure 6 Flowchart of a deep neural network training method embodiment of the present invention.

[0068] Figure 7 Flowchart of a deep neural network training method embodiment of the present invention. DETAILED DESCRIPTION

[0069] In the process of implementing the present invention, the applicants realized that due to the huge number of parameters of deep neural networks and the overly complex models, an important issue that must be considered when using them is how to improve their ability to handle parallel computing and reduce time costs and storage space. In the hardware architecture of deep neural networks, data storage and handling are the most important factors affecting the efficiency of training and reasoning:

[0070] ① During the training process, a large amount of weights, activation values, key-value caches and other data require a large memory overhead, and frequent read and write access to a large amount of memory will increase the power consumption and delay of training;

[0071] ② During the inference process, the on-chip and off-chip storage of data such as weights, activation values, and key-value caches places high demands on read and write bandwidth.

[0072] In order to solve the above problems, the present invention proposes to adopt different precisions, different processing methods, and different deployment methods for data with different amounts of information, so as to optimize the training and reasoning process without losing accuracy, reduce the computational complexity of the model and the demand for storage resources, thereby reducing power consumption and time costs.

[0073] A first aspect of the present invention provides a data deployment system for a deep neural network. Figure 1 FIG. 1 is a schematic diagram of the structure of the first embodiment of the deep neural network data deployment system of the present invention. Figure 1 As shown, the deep neural network data deployment system of this embodiment includes:

[0074] Storage module 10;

[0075] A matrix decomposition module 20, configured to decompose an original matrix into a sum of N sub-matrices, N≥2; wherein at least one of the N sub-matrices is further decomposed into a product of two or more low-rank matrices, wherein the rank of the low-rank matrix is ​​less than the rank of the original matrix;

[0076] The matrix expression processing module 30 is used for:

[0077] For the undecomposed submatrix, express it with the first precision and store it in the storage module;

[0078] For the decomposed sub-matrix, the corresponding low-rank matrix is ​​expressed with the second precision and stored in the storage module;

[0079] The original matrix is ​​a weight matrix and / or an activation value matrix and / or a key-value cache matrix of a deep neural network, and the first precision is lower than the second precision.

[0080] In this embodiment, N=2; the matrix decomposition module 20 is used to perform low-rank adaptive decomposition (LORA) on the original matrix W. Low-rank adaptive decomposition decomposes the original matrix into a main feature matrix and a low-rank matrix. The size of the main feature matrix is ​​the same as that of the original matrix, representing the main features in the pre-trained model. The low-rank matrix uses the product of two low-rank matrices to represent the features related to the current task. The "low rank" here means that the rank of the matrix is ​​less than the rank of the original matrix.

[0081] Specifically, the original matrix W is obtained by low-rank adaptive decomposition (LORA):

[0082] W = J + L × R: or

[0083]

[0084] Among them: the undecomposed submatrix is: the main characteristic matrix J, whose dimension is the same as the original matrix; the two low-rank matrices corresponding to the decomposed submatrix are: the second-first low-rank matrix L, the second-second low-rank matrix R, and the rank k of the two low-rank matrices is less than the rank of the original matrix.

[0085] Figure 2 for Figure 1 A schematic diagram showing the characteristics and corresponding storage methods of the main feature matrix and low-rank matrix in the low-rank adaptive decomposition in the deep neural network data deployment system.

[0086] In this embodiment, the storage module 10 includes: a low-frequency read-write area 11 and a high-frequency read-write area 12. It should be particularly noted that the "low frequency" and "high frequency" here are only relative concepts, that is, the low-frequency read-write area 11 has a lower read-write frequency than the high-frequency read-write area 12, and the high-frequency read-write area 12 has a higher read-write frequency than the low-frequency read-write area 11.

[0087] In the training for a specific task, the main feature matrix J does not participate in the training task, and the parameters are relatively fixed. There is no need to read frequently and occupy too much training memory during training, so the main feature matrix J is stored with lower precision, such as integer data, in the low-frequency read-write area 11 of the large memory with faster reading speed; and the two low-rank matrices L and R with greater relevance to the current task, both participate in the training task, and the parameters are updated frequently. They have high requirements for reading and writing, but occupy less storage space and use higher precision, such as floating point numbers, which are stored in the high-frequency read-write area 12 of the more expensive small memory to better complete the training process.

[0088] It should be noted that for the main feature matrix, it contains the main information of the original matrix. Even if it is stored with lower precision, the memory it occupies is still relatively large, so it is called "large memory". For the two low-rank matrices, they contain less original matrix information. Even if they are stored with higher precision, the memory they occupy is still small, so they are called "small memory".

[0089] Specifically, in the matrix expression processing module, the main feature matrix J is stored using integer data precision; in the matrix expression processing module, the second-first low-rank matrix L and the second-second low-rank matrix R are both expressed and stored using one of the following data precisions: Float32, Float64, or Float16.

[0090] Specifically, the storage module includes: a low-frequency read-write area and a high-frequency read-write area; a matrix expression processing module: storing the main feature matrix in the low-frequency read-write area of ​​the storage module; storing the second-first low-rank matrix and the second-second low-rank matrix in the high-frequency read-write area of ​​the storage module. It should be noted that the "low-frequency read-write area" and "high-frequency read-write area" in this article are only relative to another memory area, and do not represent an absolute meaning.

[0091] Specifically, the rank k of the low-rank adaptive decomposition satisfies: 1≤k≤20%D, where D is the rank of the original matrix. Preferably, the rank k of the low-rank adaptive decomposition is selected from one of the following values: 2, 3, 4, 8, 16, 32.

[0092] The matrix decomposition module of this embodiment can use various matrix decomposition methods to achieve low-rank adaptive decomposition, which is described below using singular value decomposition and eigenvalue decomposition as examples.

[0093] 1. Singular Value Decomposition

[0094] When the matrix decomposition module 20 uses the singular value decomposition method to perform low-rank adaptive decomposition on the original matrix, the following steps are performed:

[0095] (1) Perform singular value decomposition on the original matrix: W = UΣV T , where Σ is a diagonal matrix, the elements on its diagonal are singular values, and these singular values ​​are arranged in order of size;

[0096] (2) decomposing Σ into the sum of two diagonal matrices Σ=Σ1+Σ2, where the first diagonal matrix Σ1 contains the first k singular values ​​and the second diagonal matrix Σ2 contains the remaining singular values;

[0097] (3) Using the decomposed Σ, we can obtain: W = UΣV T =U(Σ1+Σ2)V T =UΣ1V T +UΣ2V T ;in:

[0098] UΣ1V T As the main feature matrix J;

[0099] U and Σ2V T As the second low-rank matrix L, the second low-rank matrix R respectively; or, UΣ2 and V T As the second-first low-rank matrix L and the second-second low-rank matrix R respectively;

[0100] In this embodiment, the sum of squares of the singular values ​​in the first diagonal matrix Σ1 accounts for more than 95% of the sum of squares of the singular values ​​in the diagonal matrix Σ.

[0101] The following specifically introduces the specific implementation method and beneficial effects of the low-rank adaptive decomposition method using singular value decomposition combined with different precision expressions. q Take (4096*4096) as an example, the value is:

[0102]

[0103] Follow these steps:

[0104] (1) Determine the rank n of the low-rank matrix to be 5% of the rank of the original matrix. The 5% here is consistent with the subsequent use of 5% of the value in the diagonal matrix Σ as the main feature matrix;

[0105] (2) The original matrix W q Perform singular value decomposition: W q =U∑V T , where Σ is a diagonal matrix, the elements on its diagonal are singular values, and these singular values ​​are arranged in order of size;

[0106] (3) decomposing Σ into the sum of two diagonal matrices Σ=Σ1+Σ2, where the first diagonal matrix Σ1 contains the first n largest decomposition values, n is the rank of the low-rank matrix set above, and the second diagonal matrix Σ2 contains the remaining singular values;

[0107] (4) UΣ1V T As the main feature matrix J; U and Σ2V T They are respectively used as the second-first low-rank matrix L and the second-second low-rank matrix R. The ranks of the two low-rank matrices are 5% of the ranks of the original matrices.

[0108] (5) The two low-rank matrices are stored using the original storage precision. The main feature matrix J is expressed using an integer 8 bits.

[0109] After the above processing is implemented, the original matrix W is restored based on the main feature matrix J and the second low-rank matrix L and the second low-rank matrix R. q ':

[0110]

[0111] Compare the original matrix W q And the restored original matrix W q 'It can be seen that W q 'Retained W q 97.1% of the data values ​​in the data. While keeping the accuracy basically unchanged, the storage space is compressed by 50%, and the execution speed is increased to 2 times the original speed.

[0112] 2. Eigenvalue Decomposition

[0113] When the matrix decomposition module 20 decomposes the original matrix using the eigenvalue decomposition method, the following steps are performed:

[0114] (1) Solve the eigenvalue diagonal matrix D of the original matrix and the corresponding eigenvector V, where D is a diagonal matrix and is arranged in descending order;

[0115] (2) Decompose the original matrix into: W = VDV T ;

[0116] (3) Decomposition of the diagonal matrix D: D = D1 + D2, and then W = V(D1 + D2)V T =VD1V T +VD2V T , where D1 retains the first k elements in the diagonal matrix D, and D2 retains the other elements in the diagonal matrix, where:

[0117] VD1V T As the main feature matrix J;

[0118] V and D2V T As the second low-rank matrix L and the second low-rank matrix R respectively; or, VD2 and V T They are respectively used as the second-first low-rank matrix L and the second-second low-rank matrix R.

[0119] The second low-rank matrix L and the second low-rank matrix R are both saved with high precision (Float16, Float32 or Float64). The main feature matrix J is saved with low precision (integer type), and the decomposition is completed.

[0120] After simulation, it is found that the eigenvalue decomposition method can achieve the same beneficial effect as the singular value decomposition, which will not be repeated here.

[0121] In this embodiment, during the training phase, the data of the main feature matrix remains unchanged. During the training phase, the data of the low-rank matrix can change as the training progresses. For low-rank matrices that need to be updated with parameters, high-precision (Float16, Float32 or Float64) memory is used to read and write to improve computing efficiency; for relatively fixed main feature matrices, a low-frequency read-write type of storage is used, and a longer time interval is used to update. Different from the traditional matrix decomposition method, this method takes into account the functional, price, and bandwidth constraints of different types of storage, so that the deployment of large language models can have both high performance, high cost performance, and small area.

[0122] At this point, the first embodiment of the deep neural network data deployment system of the present invention has been introduced.

[0123] In a second exemplary embodiment of the present invention, another deep neural network data deployment system is provided. This embodiment is similar to the first embodiment, except that: a main master weight decomposition method is used to perform matrix decomposition to obtain a main master weight matrix and a residual matrix, and compression and / or quantization are used to process the residual matrix, and the storage module 10 includes: an on-chip storage unit 13 and an off-chip storage unit 14.

[0124] Figure 3 FIG. 2 is a schematic diagram of the structure of the second embodiment of the deep neural network data deployment system of the present invention. Figure 3 As shown, the core of the sovereign weight decomposition is to decompose the original matrix into a sovereign weight low-rank matrix (obtained by multiplying two low-rank matrices) and a residual matrix: that is, we get:

[0125] W=S×T+K

[0126] Among them, the part that accounts for a large weight value or a large storage value is represented by the product of two low-rank matrices - the first low-rank matrix S and the first second low-rank matrix T, which is called the master weight matrix; the remaining residual part that cannot be expressed is obtained by subtracting the master weight matrix from the original matrix, that is, the residual matrix K is obtained, which is the same size as the original matrix. The matrix expression processing module is used to: express the first low-rank matrix S and the first second low-rank matrix T with the second precision (higher precision) and then store them; compress and / or quantize the residual matrix K, express it with the first precision (lower precision) and then store it.

[0127] Figure 4 for Figure 3 The schematic diagram of the main feature matrix and low-rank matrix in the main weight decomposition and their corresponding storage methods in the deep neural network data deployment system is shown in FIG. Figure 4 As shown, since the sovereign weight matrix has expressed most of the information, in the matrix expression processing module, the first low-rank matrix S and the first second low-rank matrix T are both stored in the on-chip storage unit 13 using Float16, Float32 or Float64 data precision.

[0128] On this basis, the required storage space can be reduced by compressing the residual matrix. The methods used can be the following two:

[0129] ① Sparse matrix compression algorithm

[0130] A highly compressed storage method for sparse matrices (lossless or lossy compression). In simple terms, what was originally stored was a matrix, and now only the sparse data in the matrix needs to be stored (and its location is recorded). This compression method is beneficial when the matrix is ​​sparse.

[0131] ②Sparse matrix quantization

[0132] The use of low-bit quantization coding, in layman's terms, is to convert the numbers originally expressed by floating-point numbers into integers. A floating-point number requires 2B (Float16), 4B (Float32) or 8B (Float64) to store, while a 4-bit integer only requires 0.5B to store. Of course, some data will be lost, but the impact on the overall performance is limited. Sparse matrix quantization is generally used in the reasoning process of neural networks, that is, compression coding is used in the off-chip storage unit 14 to achieve storage and bandwidth savings. After loading onto the chip, the decoding and decompression operations are completed on the chip to complete the reasoning.

[0133] The matrix decomposition module of this embodiment can use various matrix decomposition methods to achieve primary weight decomposition, such as singular value decomposition, eigenvalue decomposition, QR decomposition, LU decomposition, etc. The following uses singular value decomposition as an example for description.

[0134] The matrix decomposition module 20 uses singular value decomposition to perform a primary re-decomposition on the original matrix, and performs the following steps:

[0135] (1) Determine the rank k of the low-rank matrix, k ≥ 1;

[0136] (2) Determine the quantization bit width n of the weight, where n ≥ 1;

[0137] (3) Perform singular value decomposition on the original matrix; W = UΣV T ; Where Σ is a diagonal matrix, the elements on the diagonal are singular values, and these singular values ​​are arranged in order of size;

[0138] (4) Decompose Σ into the sum of two diagonal matrices: Σ = Σ1 + Σ2, where the first diagonal matrix Σ1 contains the first k singular values ​​and the second diagonal matrix Σ2 contains the remaining singular values;

[0139] (5) Using the decomposed Σ, we can obtain: W = UΣV T =U(Σ1+Σ2)V T =UΣ1VT+UΣ2V T ;in,

[0140] UΣ2V T As the residual matrix K;

[0141] Set U and Σ1V T As the first low-rank matrix S and the first second low-rank matrix T respectively; or, UΣ1 and V T As the first low-rank matrix S and the first second low-rank matrix T respectively;

[0142] (6) Take the maximum and minimum values ​​of the residual matrix K, and divide the value of the residual matrix K into two according to the quantization bit width n of the weight. nClass, respectively expressed by integers, to obtain a new residual matrix, which is compressed using a sparse compression algorithm. Among them, because there are many 0s in the sparse matrix, a sparse compression algorithm can be used to complete the compression to save storage space.

[0143] In this embodiment, the rank k of the low-rank matrix can be preferably 1 / 100 to 1 / 20 of the matrix dimension according to the compression rate, that is, if the matrix is ​​a large matrix of 4000 dimensions, its first 1% of eigenvalues ​​can be retained, and if the matrix is ​​a matrix of 100 dimensions, its first 5% can be retained, that is, the first 5 large eigenvalues. More specifically, the rank k of the low-rank matrix satisfies: Where H is the rank of the original matrix.

[0144] In this embodiment, the quantization bit width n of the weight satisfies: 2≤n≤8. Preferably, for a large language model, the quantization bit width is preferably 4, that is, 4 bits are used to represent a value of the sparse matrix.

[0145] The following specifically introduces the specific implementation method and corresponding beneficial effects of using the main weight decomposition method of singular value decomposition combined with different precision expressions. q Take (4096*4096) as an example, the value is:

[0146]

[0147] Follow these steps:

[0148] (1) Determine that the rank n of the low-rank matrix is ​​5% of the rank of the original matrix and the quantization bit width is 8.

[0149] (2) The original matrix W q Perform singular value decomposition: W q =UΣV T , where Σ is a diagonal matrix, the elements on its diagonal are singular values, and these singular values ​​are arranged in order of size;

[0150] (3) Decompose Σ into the sum of two diagonal matrices Σ = Σ1 + Σ2, where Σ1 takes the first 5% of the data in Σ and Σ2 takes the data in Σ except Σ1;

[0151] (4) Using the decomposed Σ, we can obtain: W = UΣV T =U(Σ1+Σ2)V T =UΣ1VT+UΣ2V T ; Among them, UΣ2V T As the residual matrix K; U and Σ1V T As the first low-rank matrix S and the first second low-rank matrix T respectively;

[0152] (5) The two low-rank matrices S and T are expressed using Float16 precision, and the data in the residual matrix K is encoded using 8-bit low-bit quantization.

[0153] After the above processing is implemented, the original matrix W is restored based on the two low-rank matrices S and T expressed in Float16 precision and the residual matrix encoded by 8-bit low-bit quantization q ”:

[0154]

[0155] Compare the original matrix W q And the restored original matrix W q "It can be seen that by retaining the first 5% of the data values ​​and expressing the remaining residuals with 8-bit integers, 98.5% of the data accuracy can be retained and the inference speed can be increased by 0.66 times. Among them, the improvement in inference speed is mainly achieved by reducing storage space.

[0156] Similarly, the top 2% of small decomposition values ​​are retained, the two low-rank matrices use the original storage precision, and the remaining residuals are expressed as integer 8 bits. This can retain 96.2% of the data values, while the storage space can be reduced to about 50% of the size, and the inference speed can be increased by 0.85 times.

[0157] Similarly, the first 1% of the SVD decomposition values ​​are retained, the two low-rank matrices use the original storage precision, and the remaining residuals are expressed as integer 8 bits. This can retain 92.3% of the data values ​​and increase the inference speed by 0.92 times.

[0158] The deep neural network data deployment system of this embodiment can be used in the deployment stage or in the training stage; different from other methods that place the main information of the original matrix in the residual matrix here, the present invention places the main information of the matrix in two low-rank matrices to retain accuracy. Although the residual matrix is ​​large, it uses low-bit encoding, sparse matrix compression and other methods to greatly reduce the memory space occupied by the overall network and achieve a cost-effective design. The advantage of this is that the main components are saved in high-precision data during deployment, and the residual values ​​with less impact are compressed using a sparse matrix compression method to compress the residual matrix components into a low-bit method for storage.

[0159] In a third exemplary embodiment of the present invention, another deep neural network data deployment system is provided. This embodiment is similar to the first embodiment, except that the original matrix is ​​decomposed into three sub-matrices, one of which is adaptively decomposed into two low-rank matrices by low-rank.

[0160] Figure 5Schematic diagram of the characteristics and corresponding storage methods of the main feature matrix and the low-rank matrix in the low-rank adaptive decomposition in the third embodiment of the deep neural network data deployment system of the present invention. Figure 5 As shown, in this embodiment, the matrix decomposition module is used to perform low-rank adaptive decomposition on the original matrix W to obtain:

[0161] W=J1+J2+L×R

[0162] The undecomposed submatrix is: the first main characteristic matrix J1 and the second main characteristic matrix J2, whose dimensions are the same as the original matrix; the two low-rank matrices corresponding to the decomposed submatrices are: the second-first low-rank matrix L and the second-second low-rank matrix R, and the rank k of the two low-rank matrices is smaller than the rank of the original matrix.

[0163] On this basis, the first precision includes: first-first precision; first-second precision; wherein both the first-first precision and the second precision are lower than the second precision, and the first-first precision is lower than the first-second precision. The matrix expression processing module is used to: express the first main feature matrix J1 with the first-first precision and store it; express the second main feature matrix J2 with the first-second precision and store it; express the second-first low-rank matrix L and the second-second low-rank matrix R with the second precision and store them.

[0164] The matrix decomposition module of this embodiment can use various matrix decomposition methods to achieve low-rank adaptive decomposition, and singular value decomposition is used as an example for explanation. When the matrix decomposition module 20 uses the singular value decomposition method to perform low-rank adaptive decomposition on the original matrix, the following steps are performed:

[0165] (1) Decompose the original matrix. The decomposition method can be arbitrary. For example, decompose the original matrix according to SVD (singular value decomposition) into W = UΣV T ;

[0166] (2) Σ is a diagonal matrix; sort by size. At this time, Σ can be divided into multiple parts, such as the first 1% of the data as Σ1, 1% to 10% of the data as Σ2, and the data after 10% as Σ3, Σ = Σ1 + Σ2 + Σ3.

[0167] (3) Based on the above, the original matrix is ​​decomposed into W = UΣV T =U(Σ1+Σ2+Σ3)V T =UΣ1V T

[0168] +UΣ2V T +UΣ3V T ;in:

[0169] UΣ1V TAs the first main feature matrix J1;

[0170] UΣ2V T As the second main feature matrix J2;

[0171] U and Σ3V T As the second-order matrix L and the second-order low-rank matrix R respectively; or, UΣ3 and V T As the second-first matrix L and the second-second low-rank matrix R respectively;

[0172] These three parts of data represent the decomposition of different weights of the original matrix. Generally speaking, selecting 1% of the data can express most of the valid data. According to this method, the matrix can be divided into three parts, representing the 1% with the most valid bits, the 1% to 10% with the most valid bits, and the last 90% with the least valid bits. In actual use, the first 1% needs to be retained with the highest accuracy, the middle part can retain lower precision data, and the last 90% can be discarded or saved with the lowest accuracy.

[0173] Three embodiments of the deep neural network data deployment system are given above. Based on the above three embodiments, the following aspects of the deep neural network data deployment system in the present invention are explained in detail:

[0174] 1. Number of sub-matrices

[0175] In the first and second embodiments, the original matrix is ​​decomposed into two sub-matrices, one of which is decomposed into the product of two low-rank matrices. In the third embodiment, the original matrix is ​​decomposed into three sub-matrices, the last of which is decomposed into the product of two low-rank matrices.

[0176] Based on the above embodiments, those skilled in the art can understand that the original matrix can be decomposed into the sum of N sub-matrices according to the inventive concept of the present invention, N≥2, and one of the sub-matrices can be decomposed, or two or more of the sub-matrices can be decomposed. These variations can all implement the present invention and are within the protection scope of the present invention.

[0177] 2. Number of low-rank matrices

[0178] In the first, second and third embodiments, the decomposed sub-matrices are all decomposed into the product of two low-rank matrices.

[0179] However, those skilled in the art should understand that the above is only one implementation of the present invention. In other embodiments of the present invention, in order to save storage space and / or facilitate compression, the sub-matrix can be decomposed into the form of the product of 3, 4 or more low-rank matrices, which can also implement the present invention and is also within the protection scope of the present invention.

[0180] 3. Matrix decomposition method

[0181] In the first and third embodiments, the matrix decomposition adopts low-rank adaptive decomposition, specifically singular value decomposition or eigenvalue decomposition. In the second embodiment, the matrix decomposition adopts primary weight decomposition, specifically singular value decomposition.

[0182] Based on the above embodiments, those skilled in the art can understand that various existing matrix decomposition methods can be applied to the present invention, for example, QR decomposition, LU decomposition, etc., all of which can implement the present invention and are within the protection scope of the present invention.

[0183] 4. Restoration of the original matrix

[0184] In the first, second and third embodiments, the working processes of the matrix decomposition module and the matrix expression processing module are described in detail.

[0185] Those skilled in the art should understand that in some cases of subsequent processing, it is necessary to restore the original data for processing. In this case, the deep neural network data deployment system also includes: a matrix restoration module, which is used to restore the original matrix expressed with the original precision based on the undecomposed submatrix expressed with the first precision; the low-rank matrix of the decomposed submatrix expressed with the second precision. Among them, in the method adopted by the matrix restoration module, the method of restoring precision and matrix decomposition is opposite to the previous method, which will not be described in detail here.

[0186] 5. Type of the original matrix

[0187] In the above three embodiments, the weight matrix is ​​used as an example for explanation. However, in a deep neural network, the activation value matrix, the key value cache (KV Cache) matrix and the weight matrix have the same properties, and the present invention can also be applied to achieve corresponding beneficial effects, which are also within the protection scope of the present invention.

[0188] Taking the key-value cache as an example, the key-value cache is stored in the form of a matrix in the memory. During the reasoning process, as the number of reasoning times increases, the storage space occupied by the key-value cache increases linearly. The sovereign re-decomposition method of the second embodiment of the above-mentioned deep neural network data deployment system is adopted to decompose the key-value cache into two low-rank matrices and a residual matrix. The residual matrix is ​​compressed during storage to save space, and the residual matrix is ​​restored to its original size when used to complete the reasoning process. This can greatly reduce the storage space without basically affecting the data accuracy, while significantly improving the reasoning speed.

[0189] 6. Application of raw data

[0190] In the present invention, the original matrix is ​​a weight matrix and / or an activation value matrix and / or a key-value cache data matrix obtained during the training process of a deep neural network in one of the following fields: speech recognition, natural language processing, text generation, dialogue system, translation, document summarization, image recognition, image generation, video generation, image segmentation.

[0191] In other words, the present invention can be applied to weight matrices, activation value matrices, and key-value cache data including but not limited to the above-mentioned fields, and the present invention has no limitation on the technical field of original data.

[0192] 7. The core of this invention is the storage method and location based on the matrix decomposition algorithm, not the algorithm itself

[0193] The matrix decomposition algorithm is the basis of the present invention, but the core of the present invention is to store data with different amounts of information using different precision types and in different locations, thereby achieving basically no performance loss with relatively low resource consumption.

[0194] Therefore, the present invention is not a simple algorithm solution, but an optimization method that takes into account the hardware environment of model deployment on the basis of the model algorithm and unifies the deep neural network algorithm with the model hardware architecture. It can be applied to multiple application scenarios that require large-scale computing and high complexity, such as mobile devices and embedded systems, large-scale image data processing, and edge computing, to achieve fast and accurate reasoning.

[0195] 8. Beneficial effects of the deep neural network data deployment system of the present invention

[0196] The present invention aims to reduce the energy consumption and time of deep neural network (DNN) model training and reasoning, and thus proposes a solution for model weight matrix decomposition. Under the premise of keeping the model performance basically intact, the storage space required in the training and reasoning process is effectively reduced, and the computing efficiency is improved, thereby achieving the goal of saving storage resources and speeding up the reasoning speed. In addition, the compressed model has lower energy consumption and is easier to deploy, which greatly improves the practicality and scalability of the model.

[0197] Based on the first embodiment of the deep neural network data deployment system described above, a second aspect of the present invention provides a deep neural network training method.

[0198] Application scenarios of training: In the fine-tuning training of large language models, the pre-trained model is decomposed into a main feature matrix and a low-rank matrix in the form of matrix decomposition. The main feature matrix is ​​stored in a low-frequency read-write storage area to retain the main parameters. The remaining data is expressed as the product of two low-rank matrices. During the training process, only the parameters in the two low-rank matrices are changed. After a long number of iterations, the parameters in the main matrix are updated. The advantage of this is that it can reduce the amount of calculation, reduce the computational complexity of the matrix, and reduce the requirements for communication bandwidth.

[0199] Figure 6 Flowchart of an embodiment of a deep neural network training method of the present invention. Figure 6 As shown, the deep neural network training method of this embodiment includes:

[0200] Step S602, after the first iteration is completed, the obtained result is used as the original matrix, and according to the first embodiment of the deep neural network data deployment system, the main feature matrix J, the second low-rank matrix L, and the second low-rank matrix R are stored;

[0201] Step S604, after each of the subsequent 2, ..., M-1 iterations is completed, the obtained result is used as the original matrix, the main feature matrix J is kept unchanged, and two new low-rank matrices - the second-first low-rank matrix L and the second-second low-rank matrix R are obtained by using: W = J + L × R. Under the second precision expression, the second-first low-rank matrix L and the second-second low-rank matrix R are both updated, M ≥ 3;

[0202] Step S606, after the Mth iteration is completed, the obtained result is used as the original matrix, and the matrix decomposition method of the first embodiment of the deep neural network data deployment system is used to obtain a new main feature matrix J and two low-rank matrices - the second-first low-rank matrix L and the second-second low-rank matrix R; under the first precision expression, the main feature matrix J is updated, and under the second precision expression, both the second-first low-rank matrix L and the second-second low-rank matrix R are updated.

[0203] In this embodiment, the number of iterations M can be set according to the actual scenario and hardware resources. As an example, the strategy for stopping iteration is: according to the training process, the accuracy data of the test set data is obtained. If the accuracy does not increase for 20 consecutive times (or the increase is less than 0.01%), the iteration is considered to be completed and the iteration is stopped.

[0204] In the present invention, in the deep neural network data deployment stage, the two matrices are respectively represented as the main feature matrix of the model and the low-rank specific task weight matrix, so that they can be stored separately in different storage modules, and within a preset number of iterations, the two low-rank matrices are updated in each iteration, while the main feature matrix is ​​only updated once, so that the feature matrix of the model remains basically unchanged during the training process, and only the low-rank matrix parameters are updated, which greatly improves the speed of parameter training.

[0205] Based on the second embodiment of the deep neural network data deployment system described above, the third aspect of the present invention provides a deep neural network reasoning method.

[0206] Application scenarios of reasoning: In the reasoning module of the large language model, it is divided into a compression storage module, a high-precision storage module, a calculation module, etc. For each layer of reasoning calculation, the weight matrix is ​​decomposed into a residual matrix and a low-rank matrix, and placed in the corresponding storage module. The residual matrix is ​​compressed with a lower bit. During use, the low-bit weight value and the original data are directly calculated using low-bit multiplication, and then the low-rank matrix and the original data are calculated. After the calculation is completed, the values ​​obtained by the two calculation methods are added together to calculate the output data of this layer.

[0207] Figure 7 Flowchart of an embodiment of a deep neural network training method of the present invention. Figure 7 As shown, the deep neural network reasoning method of this embodiment includes:

[0208] Step S702, before inference, taking the weight matrix as the original matrix M, decomposing the original matrix according to the second embodiment of the deep neural network data deployment system, and storing the residual matrix K, the first low-rank matrix S, and the first second low-rank matrix T;

[0209] Among them, the first low-rank matrix S and the first second low-rank matrix T are expressed using the second precision (higher precision).

[0210] Step S704, during the inference process, restoring the original matrix and receiving the matrix to be calculated, including:

[0211] Sub-step S704a, decompressing and / or dequantizing the residual matrix expressed with the first precision to restore it to the residual matrix expressed with the original precision;

[0212] Sub-step S704b, multiplying the first low-rank matrix S and the first second low-rank matrix T, and adding the multiplication result to the residual matrix expressed in the original precision to obtain the restored original matrix;

[0213] In this sub-step, the restored original matrix is ​​the original floating-point matrix. Compared with the original matrix, part of the precision is lost in the quantization process, but the impact on the result is controllable.

[0214] Step S704c, performing inference calculation using the restored original matrix M' and the matrix to be calculated.

[0215] In this embodiment, in the data deployment stage of the Deep God Network, the original matrix is ​​represented as the sum of a residual matrix and a low-rank matrix, and the low-rank matrix is ​​used to represent most of the information, and the residual matrix represents the remaining small amount of information, that is, a higher compression method can be used for storage, thereby reducing the inference time.

[0216] A fourth aspect of the present invention provides a deep neural network chip. Three embodiments of the deep neural network chip are given below.

[0217] ① In the first embodiment, the deep neural network chip implements the first, second, and third embodiments of the above deep neural network data deployment system.

[0218] ② In the second embodiment, the deep neural network chip implements the above deep neural network training method embodiment.

[0219] ③In the second embodiment, the deep neural network chip implements the above deep neural network reasoning method embodiment.

[0220] At this point, the various embodiments of the present invention have been introduced. According to the above description, those skilled in the art should have a clear understanding of the present invention.

[0221] It should be noted that, unless explicitly indicated to the contrary, the numerical parameters in the specification and claims of the present invention may be approximate values ​​and may be changed according to the content of the present invention. Specifically, all the numbers indicating the content of the composition, reaction conditions, etc. recorded in the specification and claims should be understood to be modified by the term "about" in all cases, and the meaning of the expression is to include a change of ±10% from a specific number in some embodiments.

[0222] The ordinal numbers used in the specification and claims, such as "first", "second", "third", "primary", "secondary", as well as Arabic numerals, letters, etc., to modify the corresponding elements (or steps), are intended only to clearly distinguish an element (or step) with a certain name from another element (or step) with the same name, and do not mean that the element (or step) has any ordinal number, nor do they represent the order of one element (or step) and another element (or step).

[0223] Meanwhile, unless steps are specifically described or must occur in sequence, the order of the steps is not limited to that listed above and may be changed or rearranged according to desired designs.

[0224] The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0225] The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. The physical implementation of the hardware structure includes but is not limited to physical devices, which include but are not limited to transistors, memristors, DNA computers, single-chip microcomputers, microprocessors or digital signal processors (DSPs). In addition, the present invention is not directed to any specific programming language. It should be understood that the content of the present invention can be implemented using various programming languages, and the description of specific languages ​​herein is to disclose the best implementation of the present invention.

[0226] Those skilled in the art should understand that in the claims and description of the present invention, the word "comprising" does not exclude the existence of elements (or steps) not listed in the claims. The word "a" or "an" preceding an element (or step) does not exclude the existence of multiple such elements (or steps).

[0227] For certain implementations, if they are not the key content of the present invention and are well known to ordinary technicians in the relevant technical field, they are not described in detail in the drawings or text of the specification due to space limitations. In this case, reference can be made to relevant existing technologies for understanding.

[0228] Furthermore, the above embodiments are provided merely to enable the present invention to satisfy legal requirements, and the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0229] Similarly, it should be understood that in order to simplify the present invention, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the claims, each inventive aspect lies in less than all the features of the previous single embodiment. Moreover, the embodiments can be mixed and matched with each other or with other embodiments based on design and reliability considerations, that is, the technical features in different embodiments can be freely combined to form more embodiments. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0230] The above specific embodiments provide a detailed description of the objectives, technical means and beneficial effects of the present invention. It should be understood that the purpose of the detailed description is to enable those skilled in the art to understand the present invention more clearly, and it is not used to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A deep neural network data deployment system, characterized in that: include: Storage module; A matrix decomposition module, used for decomposing an original matrix into a sum of N sub-matrices, N≥2; wherein at least one of the N sub-matrices is further decomposed into a product of two or more low-rank matrices, the rank of the low-rank matrix being less than the rank of the original matrix; Matrix expression processing module, used for: For the undecomposed submatrix, expressing it with the first precision and storing it in the storage module; For the decomposed sub-matrix, expressing the corresponding low-rank matrix with the second precision and storing it in the storage module; The original matrix is ​​a weight matrix and / or an activation value matrix of a deep neural network, and the first precision is lower than the second precision.

2. The deep neural network data deployment system according to claim 1, characterized in that: The storage module includes: a low-frequency read-write area and a high-frequency read-write area; wherein the undecomposed sub-matrix is ​​stored in the low-frequency read-write area of ​​the storage module, and the low-rank matrix corresponding to the decomposed sub-matrix is ​​stored in the high-frequency read-write area of ​​the storage module; Or, the storage module includes: an off-chip storage unit and an on-chip storage unit; wherein the undecomposed sub-matrix is ​​stored in the off-chip storage unit, and the low-rank matrix corresponding to the decomposed sub-matrix is ​​stored in the on-chip storage unit.

3. The deep neural network data deployment system according to claim 1, characterized in that: The second precision is: the original precision of the original matrix; and / or, the low-rank matrix corresponding to the decomposed submatrix is ​​updated at a higher frequency than the undecomposed submatrix; And / or, the original matrix is ​​a weight matrix and / or an activation value matrix and / or a key-value cache matrix obtained during a deep neural network training process in one of the following fields: speech recognition, natural language processing, text generation, dialogue system, translation, document summarization, image recognition, image generation, video generation, image segmentation; And / or, further comprising: a matrix restoration module, for recovering the undecomposed submatrix according to the first precision expression; The low-rank matrix of the decomposed submatrices expressed in the second precision is used to restore the original matrix expressed in the original precision.

4. The deep neural network data deployment system according to claim 1, characterized in that: N＝2； The matrix decomposition module is used to perform low-rank adaptive decomposition on the original matrix W to obtain: W = J + L × R; wherein: The undecomposed submatrix is: the main feature matrix J, whose dimension is the same as the original matrix; The two low-rank matrices corresponding to the decomposed submatrices are: a second-first low-rank matrix L and a second-second low-rank matrix R, and the ranks k of the two low-rank matrices are smaller than the rank of the original matrix; The matrix expression processing module is used to: express the main feature matrix J with the first precision and store it; express the second-first low-rank matrix L and the second-second low-rank matrix R with the second precision and store them.

5. The deep neural network data deployment system according to claim 4, characterized in that: In the matrix decomposition module, the rank k of the low-rank adaptive decomposition satisfies: 1≤k≤20%D, where D is the rank of the original matrix; And / or, the rank k of the low-rank adaptive decomposition is selected from one of the following values: 2, 3, 4, 8, 16, 32; And / or, in the matrix decomposition module, the low-rank adaptive decomposition method is one of the following decomposition methods: singular value decomposition, eigenvalue decomposition; And / or, in the matrix expression processing module, the main feature matrix J is stored using integer data precision; And / or, in the matrix expression processing module, the second-first low-rank matrix L and the second-second low-rank matrix R are both expressed using one of the following data precisions for storage: Float32 type, Float64 type, Float16 type; And / or, the storage module includes: a low-frequency read-write area and a high-frequency read-write area; the matrix expression processing module: stores the main feature matrix in the low-frequency read-write area of ​​the storage module; stores the second-first low-rank matrix and the second-second low-rank matrix in the high-frequency read-write area of ​​the storage module.

6. The deep neural network data deployment system according to claim 4, characterized in that: The low-rank adaptive decomposition method is: singular value decomposition; The matrix decomposition module is used to: Perform singular value decomposition on the original matrix: W = U∑V T , where Σ is a diagonal matrix, the elements on its diagonal are singular values, and these singular values ​​are arranged in order of size; Decompose Σ into the sum of two diagonal matrices Σ=Σ1+Σ2, where the first diagonal matrix Σ1 contains the first k singular values ​​and the second diagonal matrix Σ2 contains the remaining singular values; Using the decomposed Σ, we can get: W = UΣV T =U(Σ1+Σ2)V T =UΣ1V T +UΣ2V T ;in: UΣ1V T As the main feature matrix J; U and Σ2V T As the second low-rank matrix L, the second low-rank matrix R respectively; or, UΣ2 and V T As the second-first low-rank matrix L and the second-second low-rank matrix R respectively; The sum of squares of the singular values ​​in the first diagonal matrix Σ1 accounts for more than 95% of the sum of squares of the singular values ​​in the diagonal matrix Σ.

7. The deep neural network data deployment system according to claim 4, characterized in that: The low-rank adaptive decomposition method is: eigenvalue decomposition; The matrix decomposition module is used to: Solve the eigenvalue diagonal matrix D of the original matrix and the corresponding eigenvector V, where D is a diagonal matrix and is arranged in descending order; Decompose the original matrix into: W = VDV T ; Decompose the diagonal matrix D: D = D1 + D2, and then W = V(D1 + D2)V T =VD1V T +VD2V T , where D1 retains the first k elements in the diagonal matrix D, and D2 retains the other elements in the diagonal matrix, where: VD1V T As the main feature matrix J; V and D2V T As the second low-rank matrix L and the second low-rank matrix R respectively; or, VD2 and V T They are respectively used as the second-first low-rank matrix L and the second-second low-rank matrix R.

8. The deep neural network data deployment system according to claim 1, characterized in that: N＝3； The first precision includes: a first precision; a first second precision; wherein both the first precision and the second precision are lower than the second precision, and the first precision is lower than the first second precision; The matrix decomposition module is used to perform low-rank adaptive decomposition on the original matrix W to obtain: W = J1 + J2 + L × R; wherein the low-rank adaptive decomposition is: singular value decomposition or eigenvalue decomposition: The undecomposed sub-matrices are: a first main characteristic matrix J1 and a second main characteristic matrix J2, whose dimensions are the same as the original matrix; The two low-rank matrices corresponding to the decomposed submatrices are: a second-first low-rank matrix L and a second-second low-rank matrix R, and the ranks k of the two low-rank matrices are smaller than the rank of the original matrix; The matrix expression processing module is used to: express the first main characteristic matrix J1 with the first-first precision and store it; express the second main characteristic matrix J2 with the first-second precision and store it; express the second-first low-rank matrix L and the second-second low-rank matrix R with the second-second precision and store them.

9. The deep neural network data deployment system according to claim 1, characterized in that: N＝2； The matrix decomposition module is used to perform a master re-decomposition on the original matrix W to obtain: W = K + S × T; wherein: The undecomposed submatrix is: a residual matrix K, whose dimension is the same as the original matrix; The two low-rank matrices corresponding to the decomposed submatrices are: a first low-rank matrix S and a second low-rank matrix T, and the ranks of the two low-rank matrices are smaller than the rank of the original matrix; The matrix expression processing module is used to: compress and / or quantize the residual matrix K, express it with the first precision and then store it; express the first low-rank matrix S and the first second low-rank matrix T with the second precision and then store them.

10. The deep neural network data deployment system according to claim 9, characterized in that: In the matrix expression processing module, the first low-rank matrix S and the first second low-rank matrix T are both stored using Float32 or Float64 data precision; And / or, in the matrix expression processing module, the “expressing the residual matrix K with the first precision and then storing it” adopts at least one of the following methods: The residual matrix K is expressed with integer data precision and then stored; The residual matrix K is compressed by a sparse matrix compression method and then stored. The sparse matrix compression method is: lossless compression or lossy compression; And / or, the storage module includes: an on-chip storage unit and an off-chip storage unit: the matrix expression processing module: stores both the first-low-rank matrix S and the first-second low-rank matrix T in the on-chip storage unit; and stores the residual matrix K in the off-chip storage unit after expressing it with the first precision.

11. The deep neural network data deployment system according to claim 9, characterized in that: The matrix decomposition module is used to: Determine the rank k of the low-rank matrix, k ≥ 1; Determine the quantization bit width n of the weight, n≥1; Perform singular value decomposition on the original matrix; W = UΣV T ; Where Σ is a diagonal matrix, the elements on the diagonal are singular values, and these singular values ​​are arranged in order of size; Decompose Σ into the sum of two diagonal matrices Σ=Σ1+Σ2, where the first diagonal matrix Σ1 contains the first k singular values ​​and the second diagonal matrix Σ2 contains the remaining singular values; Using the decomposed Σ, we can get: W = UΣV T =U(Σ1+Σ2)V T =UΣ1V T +UΣ2V T ;in, UΣ2V T As the residual matrix K; Set U and Σ1V T As the first low-rank matrix S and the first second low-rank matrix T respectively; or, UΣ1 and V T As the first low-rank matrix S and the first second low-rank matrix T respectively; Take the maximum and minimum values ​​of the residual matrix K, and divide the value of the residual matrix K into 2 according to the quantization bit width n of the weight n Class, respectively expressed with integers, to obtain a new residual matrix, which is compressed using a sparse compression algorithm.

12. The deep neural network data deployment system according to claim 11, characterized in that: The rank k of the low-rank matrix satisfies: Where H is the rank of the original matrix; The quantization bit width n of the weight satisfies: 2≤n≤8.

13. A deep neural network training method, characterized in that: include: After the first iteration is completed, the obtained result is used as the original matrix, and the deep neural network data deployment system according to any one of claims 4 to 8 stores the main feature matrix J, the second low-rank matrix L, and the second low-rank matrix R; After each of the subsequent 2nd, ..., M-1 iterations is completed, the obtained result is used as the original matrix, the main feature matrix J is kept unchanged, and the matrix decomposition method in the deep neural network data deployment system is used: W = J + L × R, obtain two new low-rank matrices - the second-first low-rank matrix L and the second-second low-rank matrix R. Under the second precision expression, update the second-first low-rank matrix L and the second-second low-rank matrix R, M ≥ 3; After the Mth iteration is completed, the obtained result is used as the original matrix, and the deep neural network data deployment system described in any one of claims 4 to 8 is used to obtain a new main feature matrix J and two low-rank matrices - the second-first low-rank matrix L and the second-second low-rank matrix R; under the first precision expression, the main feature matrix J is updated, and under the second precision expression, both the second-first low-rank matrix L and the second-second low-rank matrix R are updated.

14. A deep neural network reasoning method, characterized in that: include: Before inference, the weight matrix is ​​used as the original matrix, and the deep neural network data deployment system described in any one of claims 9 to 12 stores the residual matrix, the first low-rank matrix, and the first second low-rank matrix; During the inference process, the matrix to be calculated is received; Decompressing and / or dequantizing the residual matrix expressed with the first precision to restore it to the residual matrix expressed with the original precision; Multiply the first low-rank matrix S and the second low-rank matrix T, and add the multiplication result to the residual matrix expressed in the original precision to obtain the restored original matrix; The restored original matrix and the matrix to be calculated are used for inference calculation.

15. A deep neural network chip, characterized in that: The deep neural network chip is configured to implement: A deep neural network data deployment system as claimed in any one of claims 1 to 12; And / or, the deep neural network training method as claimed in claim 13; And / or, the deep neural network reasoning method as described in claim 14.

Images