Neural network model parameter compression method and reasoning method

Abstract

The invention relates to the technical field of neural network models, in particular to a neural network model parameter compression method and an inference method. According to the method, firstly, parameter values of all layers of the neural network model are unified, that is, the parameter values of all the layers are unified to be the same, that is, all the layers have the same initial matrix, so that only the initial matrix of the parameters of one layer needs to be stored, and the parameter storage pressure is reduced. On the basis, a group of independent DoRA (Dedicated Rank Adaptation) parameters are introduced into each layer, so that the flexibility of interlayer expression is improved. DoRA parameters adopt a low-rank matrix form and are combined with amplitude factors to construct a differential compensation structure of each layer. In the model training process, only the DoRA sub-matrix and the amplitude factor of each layer are updated, so that the model performance is kept and even improved under the parameter compression background. The method is suitable for reasoning deployment of a large language model, and has the advantages of high parameter compression ratio, excellent calculation efficiency, high performance retention capability and the like.

Description

Technical Field

[0001] This invention relates to the field of neural network model technology, specifically to a method for compressing neural network model parameters and an inference method. Background Technology

[0002] Large neural network models such as BERT and GPT-3 have achieved significant performance leaps in diverse tasks. The remarkable achievements of these models are largely attributed to their massive parameter systems, but this inevitably leads to huge demands on storage and computing resources. The continuous evolution of network architectures, especially the increase in network depth, has been a major factor. ResNet, by introducing shortcut connections, successfully made training thousand-layer networks a reality, while DeepNet proposed a thousand-layer Transformer network, further improving the accuracy of deep Transformer networks. In other words, the larger the number of layers in a neural network model, the higher its accuracy. However, as the number of layers in a neural network model increases, the number of model parameters that need to be stored also increases, resulting in significant storage pressure. Existing technologies employ cross-layer parameter sharing techniques to compress model parameters and reduce storage pressure. Cross-layer parameter sharing sets the parameters of all layers of the model to be the same, so only the parameters of one layer need to be stored. While this approach alleviates the storage pressure, it reduces model performance.

[0003] In summary, existing technologies struggle to balance reducing model parameter storage pressure with ensuring model performance.

[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a neural network model parameter compression method and an inference method, which solves the problem that existing technologies struggle to simultaneously reduce model parameter storage pressure and ensure model performance.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for compressing parameters of a neural network model, comprising:

[0008] The parameter values ​​of each layer of the unified neural network model are used to obtain an initial matrix that represents the initial values ​​of each parameter of each layer. The initial matrix corresponding to each layer is the same, and one of the initial matrices is stored.

[0009] An independent set of low-rank compensation structures is introduced for the initial matrix of each layer, so that each layer introduces a corresponding set of independent low-rank submatrices, which are used to provide differentiated modeling capabilities on the basis of shared backbone parameters;

[0010] Based on the initial matrix of each layer, the initial amplitude factor of each layer is obtained;

[0011] During the training phase of the neural network model, the backbone parameters shared by all layers are kept fixed, and only the low-rank submatrix and the initial amplitude factor of each layer are updated independently, forming a parameter compensation path unique to each layer, thereby improving model performance and alleviating the performance degradation problem caused by parameter sharing.

[0012] In one implementation, the parameter values ​​of each layer of the unified neural network model are used to obtain an initial matrix representing the initial values ​​of each parameter in each layer, including:

[0013] The neural network model is pre-trained to obtain the pre-trained parameter values ​​of each layer of the neural network model;

[0014] The pre-trained parameter values ​​of one layer are shared with the other layers of the neural network model to obtain an initial matrix that represents the initial values ​​of each parameter in each layer.

[0015] In one implementation, introducing a set of independent low-rank compensation structures into the initial matrix of each layer, such that each layer introduces a corresponding set of independent low-rank submatrices, includes:

[0016] The initial matrix of each layer is reduced in rank by the number of columns to obtain the first low-rank matrix;

[0017] The initial matrix of each layer is subjected to a row count reduction process to obtain a second low-rank matrix, and the first low-rank matrix and the second low-rank matrix of each layer are used as low-rank submatrices of each layer.

[0018] In one implementation, obtaining the initial amplitude factor of each layer based on the initial matrix of each layer includes:

[0019] Determine the L2 norm of each column of the initial matrix for each layer;

[0020] Based on the L2 norm of each column, the initial amplitude factor in matrix form for each layer is obtained.

[0021] In one implementation, training the neural network model to update the low-rank submatrix of each layer and the initial magnitude factor includes:

[0022] Determine the loss function for training the neural network model, and update the low-rank submatrix and the initial amplitude factor of each layer simultaneously based on the loss function.

[0023] Secondly, embodiments of the present invention also provide a neural network model inference method, comprising:

[0024] Read one of the stored initial matrices and broadcast the initial matrix to all layers of the neural network model. Read the target submatrix and target magnitude factor of each layer. The target submatrix and target magnitude factor are respectively the low-rank submatrix and initial magnitude factor updated based on the neural network model parameter compression method described above.

[0025] Based on the initial matrix, the target submatrix, and the target amplitude factor of each layer, the target matrix of the parameters of each layer is obtained;

[0026] The neural network model is controlled to obtain inference results from the input data of the target matrix.

[0027] In one implementation, obtaining the target matrix of each layer's parameters based on the initial matrix, the target submatrix, and the target magnitude factor of each layer includes:

[0028] Determine a first target submatrix and a second target submatrix within the target submatrix. The first target submatrix has the same number of rows as the initial matrix, and the number of columns in the first target submatrix is ​​less than the number of columns in the initial matrix. The second target submatrix has the same number of columns as the initial matrix and the same number of columns as the first target submatrix.

[0029] Multiply the first target submatrix by the second target submatrix to obtain the multiplication matrix;

[0030] Add the initial matrix to the multiplication matrix to obtain the addition matrix;

[0031] Determine the L2 norm of each column of the additive matrix to obtain the column norm of the additive matrix;

[0032] Based on the addition matrix, the column norm of the addition matrix, and the target amplitude factor, the target matrix of each layer parameter is obtained.

[0033] In one implementation, the neural network model is a large language model.

[0034] Thirdly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a neural network model parameter compression program stored in the memory and executable on the processor, wherein when the processor executes the neural network model parameter compression program, it implements the steps of the neural network model parameter compression method described above.

[0035] Alternatively, the terminal device includes a memory, a processor, and a neural network model inference program stored in the memory and executable on the processor. When the processor executes the neural network model inference program, it implements the steps of the neural network model inference method described above.

[0036] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a neural network model parameter compression program, wherein when the neural network model parameter compression program is executed by a processor, the steps of the neural network model parameter compression method described above are implemented.

[0037] Alternatively, the computer-readable storage medium stores a neural network model inference program, which, when executed by a processor, implements the steps of the neural network model inference method described above.

[0038] Beneficial Effects: This invention first unifies the parameter values ​​of each layer in the neural network model, ensuring that the parameter values ​​of each layer are the same, meaning each layer has the same initial matrix. Therefore, only the initial matrix of one layer needs to be stored, reducing parameter storage pressure. Before model training, the initial matrix is ​​first reduced in rank to obtain a low-rank submatrix. During each iteration of neural network model training, the low-rank submatrix and initial amplitude factor of each layer are updated, ensuring that the final trained target submatrix and target amplitude factor of each layer match the performance of that layer, thus improving the overall performance of the model. From the above analysis, it is clear that this invention can both reduce the storage pressure of model parameters and ensure model performance. Attached Figure Description

[0039] Figure 1 This is an overall flowchart of the present invention;

[0040] Figure 2 This is a schematic diagram of model training in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of parameter sharing in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the model inference stage in an embodiment of the present invention;

[0043] Figure 5This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0045] Research has shown that large neural network models such as BERT and GPT-3 have achieved significant performance leaps across diverse tasks. The remarkable achievements of these models are largely attributed to their massive parameter systems, but this inevitably leads to huge demands on storage and computing resources. The continuous evolution of network architecture, especially the increase in network depth, has been a major factor. ResNet, by introducing shortcut connections, successfully made training thousand-layer networks a reality, while DeepNet proposed a thousand-layer Transformer network, further improving the accuracy of deep Transformer networks. In other words, the larger the number of layers in a neural network model, the higher its accuracy. However, as the number of layers in a neural network model increases, the number of model parameters that need to be stored also increases, resulting in significant storage pressure. Existing technologies employ cross-layer parameter sharing techniques to compress model parameters and reduce storage pressure. Cross-layer parameter sharing sets the parameters of all layers of the model to be the same, so only the parameters of one layer need to be stored. While this approach alleviates the storage pressure, it reduces model performance.

[0046] To address the aforementioned technical problems, this invention provides a method for compressing neural network model parameters and an inference method, resolving the difficulty in simultaneously reducing model parameter storage pressure and ensuring model performance in existing technologies. Specifically, the method first unifies the parameter values ​​of each layer of the neural network model to obtain an initial matrix representing the initial values ​​of each parameter in each layer, and stores one of these initial matrices. Then, the initial matrices of each layer are reduced in rank to obtain low-rank submatrices for each layer. Next, based on the initial matrices of each layer, the initial amplitude factor for each layer is obtained. Finally, the neural network model is trained to update the low-rank submatrices and initial amplitude factors of each layer, obtaining the target submatrices and target amplitude factors for each layer. These target submatrices and target amplitude factors, along with the initial matrices, are stored. The stored target submatrices, target amplitude factors, and initial matrices are used for inference in the neural network model.

[0047] For example, a deep neural network model like the Transformer network might consist of a thousand layers, each with the same number of parameters, say 1,048,675 parameters. To ensure these parameters are consistent across all thousand layers, a thousand initial matrices are used to represent them. These initial matrices are identical, and only one of them is stored for model inference.

[0048] The rank of each of the one thousand initial matrices is reduced to obtain one thousand sets of low-rank sub-matrices. Each set of low-rank sub-matrices includes two low-dimensional matrices, the dimensions of which are smaller than the dimensions of the initial matrices.

[0049] At the same time, an initial amplitude factor is set to enable the low-rank submatrix to be restored to the initial matrix.

[0050] When training a neural network model, the low-rank submatrix of each layer and the initial magnitude factor of each layer are updated simultaneously. When training is complete, the target submatrix and the target magnitude factor of each layer can be obtained.

[0051] When inference is needed using the model, the stored initial matrix is ​​broadcast to all layers of the model, and the target submatrix and target magnitude factor for each layer are loaded into that layer of the model. Inference can then be performed using the model; inference is the model outputting the corresponding result based on the input data.

[0052] The neural network model parameter compression method of this embodiment can be applied to terminal devices. In this embodiment, such as... Figure 1 As shown, the neural network model inference method specifically includes the following steps:

[0053] S100, unify the parameter values ​​of each layer of the neural network model to obtain an initial matrix used to characterize the initial values ​​of each parameter of each layer. The initial matrices corresponding to each layer are the same, and one of the initial matrices is stored.

[0054] S200, introduce a set of independent low-rank compensation structures for the initial matrix of each layer, so that each layer introduces a corresponding set of independent low-rank sub-matrices, which are used to provide differentiated modeling capabilities on the basis of shared backbone parameters.

[0055] When a low-rank submatrix is ​​used to provide differentiated modeling capabilities based on shared backbone parameters (i.e., when each layer shares one of the initial matrices in the memory), the introduction of the low-rank submatrix ensures that the parameters of each layer still exhibit some differentiation. One of the initial matrices in the shared memory is the shared backbone parameter.

[0056] S300, based on the initial matrix of each layer, obtain the initial amplitude factor of each layer.

[0057] S400, during the training phase of the neural network model, the backbone parameters shared by all layers are kept fixed, and only the low-rank submatrix and the initial amplitude factor of each layer are updated independently to form a parameter compensation path unique to each layer.

[0058] The neural network model in steps S100 to S400 is a large language model, which is used to classify language texts or to generate text content that matches a given topic.

[0059] In this embodiment, step S100 includes the following specific steps: pre-training the neural network model to obtain the pre-training parameter values ​​of each layer of the neural network model; sharing the pre-training parameter values ​​of one layer among the pre-training parameter values ​​of each layer with the other layers of the neural network model to obtain an initial matrix used to characterize the initial values ​​of each parameter of each layer.

[0060] In other words, the model is pre-trained. After pre-training, the parameter values ​​of each layer may be different. The pre-trained parameter values ​​of one layer are shared with other layers to ensure that the pre-trained parameter values ​​of all layers are the same. Subsequently, only the pre-trained parameter values ​​of one layer need to be stored, which means that only an initial matrix needs to be stored.

[0061] In this embodiment, step S200 includes the following specific steps: Figure 2 As shown, the initial matrix V of each layer is subjected to a column reduction process to obtain a first low-rank matrix A; the initial matrix of each layer is subjected to a row reduction process to obtain a second low-rank matrix B, and the first low-rank matrix A and the second low-rank matrix B of each layer are used as low-rank submatrices of each layer.

[0062] V∈R d×k That is, the initial matrix V has d rows and k columns; A∈R d×r That is, the first low-rank matrix A has d rows and r columns; B∈R r×k That is, the second low-rank matrix B has r rows and k columns; and r is less than k, so that the dimension of A is less than the dimension of V, and the dimension of B is also less than the dimension of V.

[0063] In this embodiment, step S300 includes the following specific steps: determining the L2 norm of each column of the initial matrix V for each layer; and obtaining the initial amplitude factor m0∈R in matrix form for each layer based on the L2 norm of each column. 1×k .

[0064] Let m0 be a matrix, and let the initial matrix V be as follows:

[0065]

[0066] in, Let be the L2 norm of the j-th column of the initial matrix V. In this embodiment, the 'L2 norm' refers to the square root of the sum of the squares of the elements in a column vector.

[0067] for example

[0068] m0 = [m 11 m 12 Λm 1k ]

[0069] in,

[0070] An example illustrates the detailed process of steps S100 to S300: Figure 3 As shown, the BERT model is used in the SST-2 sentiment classification task as an example. One training method is to first use a BERT model pre-trained in the downstream task, and take the first layer (or any layer) of the BERT encoder as the shared parameters (i.e., the initial matrix).

[0071] Initialize the DoRA parameters (i.e., ...), and then continue training the BERT model until the accuracy reaches a specified level or the number of parameter updates reaches a pre-set limit. When BERT model inference is needed, the DoRA parameters plus the shared parameters are the final model parameters. The BERT model performs the sentiment classification task based on the final model parameters.

[0072] In this embodiment, step S400, training the neural network model to update the low-rank submatrix and the initial amplitude factor of each layer, includes: determining the loss function when training the neural network model, and updating the low-rank submatrix and the initial amplitude factor of each layer simultaneously according to the loss function.

[0073] Data is input into a neural network model, and the model outputs an inference result. This inference result is compared with the true label result corresponding to the input data to obtain a loss function. Based on the loss function, backpropagation is used to update A, B (A and B are low-rank sub-matrices) and m0 of each layer until training is complete, resulting in the target sub-matrix and the target amplitude factor m′0 (m′0∈R) for each layer. 1×k The target submatrix includes the first target submatrix A′ (A′∈R). d×r ) and the second objective submatrix B′ (B′∈R r×k ).

[0074] Example 2, based on Example 1, provides a neural network model inference method, including the following specific steps S501-S506:

[0075] S501, read one of the stored initial matrices V and broadcast the initial matrix to all layers of the neural network model, read the first target submatrix A′ and the second target submatrix B′ of each layer and the target amplitude factor m′0.

[0076] When using a neural network model for inference, only one V is read from memory and broadcast to all layers to reduce data transfer volume. A′, B′, and m′0 of each layer are then read to their respective layers.

[0077] S502, multiply the first target submatrix A′ by the second target submatrix B′ to obtain the multiplication matrix A′B′.

[0078] S503, add the multiplication matrix A′B′ to the initial matrix V to obtain the addition matrix V+A′B′.

[0079] S504, determine the L2 norm of each column of the additive matrix V+A′B′, and obtain the column norm of the additive matrix ||V+A′B′||, where V+A′B′∈R d×k ,‖‖V+A′B′‖‖∈R 1×k .

[0080] S505, based on the addition matrix V+A′B′, the column norm of the addition matrix ||V+A′B′||, and the target amplitude factor m′0, the target matrix V′ for each layer parameter is obtained, where V′∈R. d×k :

[0081]

[0082] In the formula, Take the reciprocal of all elements in ||V+A′B′||. For m′0 and The Hada code product, where ο is the symbol for the Hada code product. for The Hada code product with V+A′B′. Because The dimension is R 1×k The dimension of V+A′B′ is R. d×k That is The dimensions of V+A′B′ are different, therefore the Hada code product of the two cannot be directly performed. Expand to line d, that is Since all rows of elements are identical, we can then calculate... The Hada code product of V+A′B′.

[0083] S506, control the neural network model to obtain inference results using the target matrix V′ on the input data of the neural network model.

[0084] That is, after the neural network model is loaded with V′, such as Figure 4 As shown, when data x is input into a neural network model, the model will multiply the initial matrix V, the first target submatrix A′ by the second target submatrix B′, and M. The corresponding result will be output under the action.

[0085] In summary, this invention significantly reduces the number of model parameters by using cross-layer parameter sharing technology, and mitigates the performance degradation caused by cross-layer parameter sharing technology through hierarchical customized DoRA technology.

[0086] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 5 As shown, the terminal device includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a neural network model parameter compression method. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0087] Those skilled in the art will understand that Figure 5 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0088] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a neural network model parameter compression method program stored in the memory and executable on the processor. When the processor executes the neural network model parameter compression method program, it implements the following operation instructions:

[0089] The parameter values ​​of each layer of the unified neural network model are used to obtain an initial matrix that represents the initial values ​​of each parameter of each layer. The initial matrix corresponding to each layer is the same, and one of the initial matrices is stored.

[0090] An independent set of low-rank compensation structures is introduced for the initial matrix of each layer, so that each layer introduces a corresponding set of independent low-rank submatrices, which are used to provide differentiated modeling capabilities on the basis of shared backbone parameters;

[0091] Based on the initial matrix of each layer, the initial amplitude factor of each layer is obtained;

[0092] During the training phase of the neural network model, the backbone parameters shared by all layers are kept fixed, and only the low-rank submatrix and the initial amplitude factor of each layer are updated independently, forming a parameter compensation path unique to each layer.

[0093] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for compressing parameters of a neural network model, characterized in that, include: The parameter values ​​of each layer of the unified neural network model are used to obtain an initial matrix that represents the initial values ​​of each parameter of each layer. The initial matrix corresponding to each layer is the same, and one of the initial matrices is stored. An independent set of low-rank compensation structures is introduced for the initial matrix of each layer, so that each layer introduces a corresponding set of independent low-rank submatrices, which are used to provide differentiated modeling capabilities on the basis of shared backbone parameters; Based on the initial matrix of each layer, the initial amplitude factor of each layer is obtained; During the training phase of the neural network model, the backbone parameters shared by all layers are kept fixed, and only the low-rank submatrix and the initial amplitude factor of each layer are updated independently, forming a parameter compensation path unique to each layer.

2. The neural network model parameter compression method as described in claim 1, characterized in that, The parameter values ​​of each layer of the unified neural network model are used to obtain an initial matrix representing the initial values ​​of each parameter in each layer, including: The neural network model is pre-trained to obtain the pre-trained parameter values ​​of each layer of the neural network model; The pre-trained parameter values ​​of one layer are shared with the other layers of the neural network model to obtain an initial matrix that represents the initial values ​​of each parameter in each layer.

3. The neural network model parameter compression method as described in claim 1, characterized in that, The introduction of an independent set of low-rank compensation structures into the initial matrix of each layer, such that each layer introduces a corresponding independent set of low-rank submatrices, includes: The initial matrix of each layer is reduced in rank by the number of columns to obtain the first low-rank matrix; The initial matrix of each layer is subjected to a row count reduction process to obtain a second low-rank matrix, and the first low-rank matrix and the second low-rank matrix of each layer are used as low-rank submatrices of each layer.

4. The neural network model parameter compression method as described in claim 1, characterized in that, The process of obtaining the initial amplitude factor for each layer based on the initial matrix of each layer includes: Determine the L2 norm of each column of the initial matrix for each layer; Based on the L2 norm of each column, the initial amplitude factor in matrix form for each layer is obtained.

5. The neural network model parameter compression method as described in claim 1, characterized in that, The low-rank submatrix of each layer is updated independently with respect to the initial magnitude factor, including: Determine the loss function for training the neural network model, and update the low-rank submatrix and the initial amplitude factor of each layer simultaneously based on the loss function.

6. A neural network model inference method, characterized in that, include: Read one of the stored initial matrices and broadcast the initial matrix to all layers of the neural network model. Read the target submatrix and target magnitude factor of each layer, wherein the target submatrix and the target magnitude factor are respectively the low-rank submatrix and the initial magnitude factor after being updated based on the neural network model parameter compression method as described in claim 1. Based on the initial matrix, the target submatrix, and the target amplitude factor of each layer, the target matrix of the parameters of each layer is obtained; The neural network model is controlled to obtain inference results from the input data of the target matrix.

7. The neural network model inference method as described in claim 6, characterized in that, The process of obtaining the target matrix for each layer's parameters based on the initial matrix, the target submatrix, and the target amplitude factor for each layer includes: Determine a first target submatrix and a second target submatrix within the target submatrix. The first target submatrix has the same number of rows as the initial matrix, and the number of columns in the first target submatrix is ​​less than the number of columns in the initial matrix. The second target submatrix has the same number of columns as the initial matrix and the same number of columns as the first target submatrix. Multiply the first target submatrix by the second target submatrix to obtain the multiplication matrix; Add the initial matrix to the multiplication matrix to obtain the addition matrix; Determine the L2 norm of each column of the additive matrix to obtain the column norm of the additive matrix; Based on the addition matrix, the column norm of the addition matrix, and the target amplitude factor, the target matrix of each layer parameter is obtained.

8. The neural network model inference method as described in claim 6 or 7, characterized in that, The neural network model is a large language model.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a neural network model parameter compression program stored in the memory and executable on the processor. When the processor executes the neural network model parameter compression program, it implements the steps of the neural network model parameter compression method as described in any one of claims 1-5. Alternatively, the terminal device includes a memory, a processor, and a neural network model inference program stored in the memory and executable on the processor. When the processor executes the neural network model inference program, it implements the steps of the neural network model inference method as described in any one of claims 6-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a neural network model parameter compression program, which, when executed by a processor, implements the steps of the neural network model parameter compression method as described in any one of claims 1-5. Alternatively, the computer-readable storage medium stores a neural network model inference program, which, when executed by a processor, implements the steps of the neural network model inference method as described in any one of claims 6-8.

Images