A large model inference method and device based on quantized perception fine-tuning and a medium

Abstract

The application discloses a large model inference method and device based on quantized perception fine-tuning and a medium. The method comprises the following steps: performing structural modification on an original parameter matrix of a large model, determining a horizontal vectorization vector, a vertical vectorization vector and a low-bit fixed matrix corresponding to each channel original parameter matrix of the large model; performing parameter quantization fine-tuning pre-training on the large model layer by layer based on the horizontal vectorization vector, the vertical vectorization vector and the low-bit fixed matrix, and obtaining horizontal vectorization vector values and vertical vectorization vector values of each channel of the large model; determining a deployment parameter matrix of the large model for deployment and inference according to the horizontal vectorization vector values, the vertical vectorization vector values and the low-bit fixed matrix of each channel of the large model; and performing inference analysis on input data by using the large model deployed with the deployment parameter matrix to obtain an inference result of the input data.

Description

Technical Field

[0001] This invention relates to the field of large model accelerated inference technology, and more specifically, to a large model inference method, apparatus and medium based on quantization-perception fine-tuning. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the continuous breakthroughs in the field of Natural Language Processing (NLP), generative AI technology has become an important force driving industry innovation and transformation. Traditional generative models have made significant progress in text generation, but they are often limited by the massive model size and computing power requirements. In particular, considering that LLMs have billions or even trillions of parameters, the hardware requirements for training and deployment are extremely high, making it difficult to deploy and apply them privately in the process of digital transformation of government and enterprises. From a security perspective, government and enterprise customers often find it difficult to carry out training and application of large models through SaaS service models, resulting in a vicious cycle where the better the model's performance and the larger its scale, the more difficult it is to deploy and apply it.

[0003] To address the challenges of large model parameter sizes, resource-intensive training and fine-tuning, and difficulties in private deployment, three main methods exist: quantization, pruning, and distillation. Quantization is a powerful method for compressing and accelerating neural networks by discretizing parameters into low-bit integers (e.g., converting Float16 to int8) while maintaining a shared high-precision scale within each parameter group (e.g., channels or layers). However, quantization requires very low efficiency in updating all parameters during training. Large model pruning effectively reduces the number of model parameters and computational load by accurately identifying and removing parameters or connections that contribute little to performance, allowing large models to achieve the same results with fewer parameters and connection layers. However, pruning may cause slight performance degradation, requiring precise evaluation and fine-tuning after pruning, often necessitating significant human resources. Large model distillation uses connection layer merging or parameter synchronization to achieve rapid training or fine-tuning with fewer or more connection layers and the same parameters, often at the cost of some accuracy. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a large-scale model inference method, apparatus, and medium based on quantization-based perception fine-tuning.

[0005] According to one aspect of the present invention, a large model inference method based on quantization-aware fine-tuning is provided, comprising:

[0006] The original parameter matrix of the large model is structurally modified to determine the horizontal vectorized vector, vertical vectorized vector, and low-bit fixed matrix corresponding to the original parameter matrix of each channel.

[0007] Based on the horizontal vectorized vector, the vertical vectorized vector, and the low-bit fixed matrix, the large model is pre-trained by performing parameter quantization fine-tuning layer by layer to obtain the horizontal vectorized vector values ​​and the vertical vectorized vector values ​​of each channel of the large model.

[0008] Based on the horizontal vectorized vector values, vertical vectorized vector values, and low-bit fixed matrix of each channel of the large model, determine the deployment parameter matrix for the deployment inference of the large model;

[0009] A large model with a deployment parameter matrix is ​​used to perform inference analysis on the input data to obtain the inference results.

[0010] Optionally, the original parameter matrix of the large model is structurally modified to determine the horizontal vectorized vector, vertical vectorized vector, and low-bit fixed matrix corresponding to the original parameter matrix of each channel, including:

[0011] The high-precision original parameter matrix W of each channel of the large model is decomposed horizontally and vertically to obtain a high-precision horizontal vectorized vector T0, a high-precision vertical vectorized vector S0, and a low-bit fixed matrix W0', where the original parameter matrix W of each channel is approximately S0 * W0'.

[0012] Optionally, the bit width of the parameter quantization fine-tuning pre-training is 4.

[0013] Optionally, based on the horizontal vectorized vectors, vertical vectorized vectors, and low-bit fixed matrices, the large model is pre-trained layer by layer using parameter quantization fine-tuning to obtain the horizontal and vertical vectorized vector values ​​of each channel of the large model, including:

[0014] Based on the vertical vectorization vector of the first layer channel and the low-bit fixed matrix, the first layer channel of the large model is pre-trained for quantization fine-tuning to obtain the first layer quantization result.

[0015] Based on the vertical vectorization vector and low-bit fixed matrix of the next layer channel, the next layer channel of the large model is pre-trained for quantization fine-tuning according to the first layer quantization result and the horizontal vectorization vector to obtain the next quantization result.

[0016] The iteration uses the quantization results of the previous layer channel of the large model to perform quantization fine-tuning pre-training on the next layer channel until the last layer channel of the large model is reached, thus obtaining the horizontal and vertical vectorized vector values ​​of each channel of the large model.

[0017] Optionally, the quantization expression for fine-tuning the pre-trained current layer channels of a large model is:

[0018]

[0019] The quantization expression for the next layer channel of the large model's quantization fine-tuning pre-training is as follows:

[0020] W″0=(T0+ΔT)*W

[0021] In the formula, W is the quantization result of the current channel; W″0 is the quantization result of the next channel of the current channel; S0 is the vertical quantization vector; T0 is the horizontal quantization vector; ΔS represents the gradient update of S0 obtained through fine-tuning training; ΔT represents the gradient update of T0 obtained through fine-tuning training; z0 represents an adjustment parameter; and clamp represents a function that restricts the value to [a, b].

[0022] According to another aspect of the present invention, a large model inference apparatus based on quantization-aware fine-tuning is provided, comprising:

[0023] The structural modification module is used to modify the original parameter matrix of the large model and determine the horizontal vectorized vector, vertical vectorized vector, and low-bit fixed matrix corresponding to the original parameter matrix of each channel.

[0024] The pre-training module is used to perform layer-by-layer parameter quantization fine-tuning pre-training on a large model based on horizontal vectorized vectors, vertical vectorized vectors, and low-bit fixed matrices, and to obtain the horizontal vectorized vector values ​​and vertical vectorized vector values ​​of each channel of the large model.

[0025] The determination module is used to determine the deployment parameter matrix for deployment inference of the large model based on the horizontal vectorized vector values, vertical vectorized vector values, and low-bit fixed matrix of each channel of the large model.

[0026] The inference module is used to perform inference analysis on the input data using a large model with deployment parameter matrices to obtain the inference results of the input data.

[0027] Optionally, the structural modification module includes:

[0028] The decomposition submodule is used to perform horizontal and vertical decomposition on the high-precision original parameter matrix W of each channel of the large model, resulting in a high-precision horizontal vectorized vector T0, a high-precision vertical vectorized vector S0, and a low-bit fixed matrix W0', where the original parameter matrix W of each channel is approximately S0 * W0'.

[0029] Optionally, the bit width of the parameter quantization fine-tuning pre-training is 4.

[0030] Optionally, the pre-trained module includes:

[0031] The first pre-training submodule is used to perform quantization fine-tuning pre-training on the first-layer channels of a large model based on the vertical vectorization vector of the first-layer channel and the low-bit fixed matrix, and to obtain the first-layer quantization result.

[0032] The second pre-training submodule is used to perform quantization fine-tuning pre-training on the next layer channel of the large model based on the vertical vectorization vector and low-bit fixed matrix of the next layer channel, according to the first layer quantization result and the horizontal vectorization vector, to obtain the next quantization result.

[0033] The three pre-training submodules are used to iteratively perform quantization fine-tuning pre-training on the next layer channel by using the quantization results of the previous layer channel of the large model, until the last layer channel of the large model, to obtain the horizontal vectorized vector value and the vertical vectorized vector value of each channel of the large model.

[0034] Optionally, the quantization expression for fine-tuning the pre-trained current layer channels of a large model is:

[0035]

[0036] The quantization expression for the next layer channel of the large model's quantization fine-tuning pre-training is as follows:

[0037] W0=(T0+ΔT)*W

[0038] In the formula, W is the quantization result of the current channel; W″0 is the quantization result of the next channel of the current channel; S0 is the vertical quantization vector; T0 is the horizontal quantization vector; ΔS represents the gradient update of S0 obtained through fine-tuning training; ΔT represents the gradient update of T0 obtained through fine-tuning training; z0 represents an adjustment parameter; and clamp represents a function that restricts the value to [a, b].

[0039] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described in any of the above aspects of the present invention.

[0040] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the preceding aspects of the present invention.

[0041] The method for accelerating large model inference based on quantization-aware fine-tuning provided by this invention has the following beneficial effects:

[0042] 1. This method inherits the advantages of quantization of large models and distillation acceleration, reduces memory consumption during training and deployment, and enables ultra-large-scale language models to be deployed and inferred with very few hardware resources, which can be quickly implemented in government and enterprise digital transformation projects.

[0043] 2. Simultaneously, the reduction in model size due to quantization reduces memory consumption during deployment. By reducing the number of memory accesses, inference speed during deployment is improved.

[0044] 3. At low precision, it can basically maintain the model's capabilities in language modeling, learning and understanding contexts with a small number of samples, achieving performance comparable to high-precision models. This avoids the drawback of requiring a large amount of manpower for adjustments after fine-tuning, as is the case with pruning methods. Attached Figure Description

[0045] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures:

[0046] Figure 1 This is a flowchart illustrating a large model inference method based on quantization-perception fine-tuning provided in an exemplary embodiment of the present invention.

[0047] Figure 2 This is another flowchart illustrating a large model inference method based on quantization-perception fine-tuning provided in an exemplary embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the original parameter matrix of a large model provided in an exemplary embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of structural modification in a large model inference method based on quantization-perception fine-tuning provided by an exemplary embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the quantization fine-tuning of a large model inference method based on quantization perception fine-tuning provided in an exemplary embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram illustrating the deployment and inference of a large model inference method based on quantization-aware fine-tuning provided in an exemplary embodiment of the present invention.

[0052] Figure 7 This is a schematic diagram of the structure of a large model inference device based on quantization-sensory fine-tuning provided in an exemplary embodiment of the present invention;

[0053] Figure 8 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0054] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0055] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0056] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0057] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0058] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.

[0059] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0060] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0061] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0062] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0063] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0064] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0065] The embodiments of this invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Well-known examples of terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0066] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0067] Exemplary methods

[0068] Figure 1 This is a flowchart illustrating a large model inference method based on quantization-aware fine-tuning provided in an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, the large model inference method 100 based on quantization-perception fine-tuning includes the following steps:

[0069] Step 101: Modify the structure of the original parameter matrix of the large model to determine the horizontal vectorization vector, vertical vectorization vector and low-bit fixed matrix corresponding to the original parameter matrix of each channel.

[0070] Step 102: Based on the horizontal vectorized vector, the vertical vectorized vector, and the low-bit fixed matrix, perform parameter quantization fine-tuning pre-training on the large model layer by layer to obtain the horizontal vectorized vector values ​​and vertical vectorized vector values ​​of each channel of the large model.

[0071] Step 103: Determine the deployment parameter matrix for deployment inference of the large model based on the horizontal vectorized vector values, vertical vectorized vector values, and low-bit fixed matrix of each channel of the large model.

[0072] Step 104: Use a large model with a deployment parameter matrix to perform inference analysis on the input data and obtain the inference results of the input data.

[0073] Specifically, the purpose of this invention is to provide an inference acceleration method based on quantization-aware fine-tuning. Building upon the traditional quantization compression structure and incorporating model distillation, this method decomposes the parameter matrix of each fully connected layer into a low-bit fixed matrix and a quantization vector. During inference tasks, only the quantization vector is fine-tuned, while the fixed matrix remains unchanged. For large quantized models, only the quantization vector is updated, using fewer trainable parameters. Efficient task-specific parameter storage and rapid switching reduce DRAM usage during training and deployment, and inference acceleration is achieved through reduced memory access during deployment. This addresses the challenge of private deployment due to insufficient hardware resources during digital transformation in government and enterprise sectors, meeting practical application needs.

[0074] The overall solution of this invention can be divided into three stages: structural modification, quantitative fine-tuning, and deployment reasoning.

[0075] The structural modification aims to alter the original fine-tuning training structure, comprising two steps: pre-quantization and post-quantization. First, pre-quantization transforms the parameter matrix into a quantized vector and a fixed matrix. The parameters of the fixed matrix are discretized into low-bit integers, while maintaining a shared high-precision scale within each parameter group (e.g., channel or layer), thereby compressing and accelerating the neural network. Then, post-quantization incorporates the perceptual quantized vector portion during each fine-tuning process. See the detailed diagram for further explanation. Figure 2 As shown.

[0076] The original parameter matrix is ​​the training parameter matrix of the model itself, denoted by W0, such as... Figure 3 As shown.

[0077] Furthermore, such as Figure 4 As shown, the structural modification process decomposes the original parameter matrix into a horizontal quantization vector T0 (also called a horizontal distillation vector) and a vertical quantization vector S0. The decomposed original parameter matrix becomes a low-bit fixed matrix W0', and the parameters of the subsequent fixed matrix remain unchanged. The original matrix is ​​the product of the vertical quantization vector and the low-bit fixed matrix, i.e., W0≈S0*W0'. After obtaining an approximate original parameter matrix W0 through the low-bit fixed matrix W0', the original parameter matrices W0 between each parameter channel are transformed using the horizontal quantization vector T0. In this way, the original large number of training parameters (k channels * a rows * b columns) can be reduced to a small number of parameters (horizontal quantization vector * vertical quantization vector) to achieve the same effect.

[0078] Furthermore, in the quantization fine-tuning stage, for each fully connected layer's pre-trained original parameter matrix W0, given a bit width b (set to 4 in this invention), the pre-trained weight matrix W can be expressed as (this weight matrix is ​​a general formula; in actual calculation, the matrix W of the first channel is calculated first, and then the weight matrices W of the remaining channels are calculated based on W and T0):

[0079]

[0080] W″0=T0*W=s0*T0*W

[0081] Where W0” is the matrix of other channels, and · represents the product of two matrices. The function represents the rounding function. `clamp` represents a function that restricts values ​​to the range [a, b], meaning the values ​​in this patent are within the range [0, 15]. The scale and zero point for each channel are defined. `z0` represents an adjustment parameter; when `z0` is 0, no adjustment is made. For simplified calculation, this patent uses 0. Furthermore, `S0` represents the vertical vector, and `T0` represents the horizontal vector. The final weight matrix is ​​obtained by multiplying the changing horizontal and vertical matrices with a fixed vector matrix, as shown below. Figure 5 As shown.

[0082] W'0 is the integer quantization index of W0, applicable to every fully connected layer in a pre-trained large language model. We then fine-tune only Sx and Tx (outside the clamp function in the formula), while sharing W0' across all downstream tasks (all changes are within Sx). x and T x (Changes are made above). Therefore, the quantized pre-trained weights W0 are adapted to the downstream task as follows:

[0083]

[0084] W″0=(T0+ΔT)*W

[0085] Here, ΔS represents the gradient update of S0 obtained through fine-tuning training, and ΔT represents the gradient update of T0 obtained through fine-tuning training. This method is a memory-efficient fine-tuning method specifically designed for quantizing large language models. It combines the large model distillation approach, updating only the quantization vector S0 and the distillation vector T0. Since W0' is a fixed matrix, it is frozen and shared across all training channels. When switching to different downstream tasks, the values ​​of S0 and T0 can be quickly changed. Therefore, S0 mainly updates the weight matrix of the current channel, while T0 updates the weight matrix across channels. With the fixed matrix remaining unchanged and existing only once, updating only the horizontal and vertical weights is sufficient to affect the weight matrix parameters of all channels.

[0086] Furthermore, such as Figure 6As shown, during the deployment and inference phases, the differences between channels are expressed by adjusting the values ​​of the horizontal and vertical quantization vectors Sx and Tx. For the fine-tuning task of each channel, only the horizontal and vertical quantization vectors are finely tuned (i.e., only the values ​​of Tx and Sx are updated), while the integer fixed matrix remains unchanged. For large-scale language models after quantization, the quantization awareness capability is preserved while reducing the computational cost per channel. Simultaneously, the distillation and compression approach reduces the computational cost of updating cross-layer vector matrices, employs fewer trainable parameters, and features efficient task-specific parameter storage and fast switching. Furthermore, the benefits of quantization are evident during deployment and inference, resulting in reduced DRAM usage during training and deployment (from the original N vector matrices of Float16 or higher precision to a fixed 1 vector matrix of int4), and accelerated inference due to reduced memory access during deployment.

[0087] The key technical point of this invention is:

[0088] 1. It introduces two main approaches: quantization and distillation. The traditional update of high-precision feature (vector) matrices is transformed into the update of vertical quantization vectors and horizontal distillation vectors, which greatly reduces the resource consumption during deployment, inference and fine-tuning.

[0089] 2. This method inherits the advantages of quantization for large models and distillation for acceleration. It reduces memory consumption during training and deployment by updating only the quantization vector, maintaining the frozen state of the fixed matrix, thereby reducing memory usage during training.

[0090] 3. Simultaneously, the reduction in model size due to quantization reduces memory consumption during deployment. By reducing the number of memory accesses, inference speed during deployment is improved.

[0091] 4. At low precision, it can maintain the model's ability in language modeling, learning and understanding situations with a small number of samples, achieving performance comparable to or better than the full-precision model.

[0092] The method for accelerating large model inference based on quantization-aware fine-tuning provided by this invention has the following advantages:

[0093] 1. This method inherits the advantages of quantization of large models and distillation acceleration, reduces memory consumption during training and deployment, and enables ultra-large-scale language models to be deployed and inferred with very few hardware resources, which can be quickly implemented in government and enterprise digital transformation projects.

[0094] 2. Simultaneously, the reduction in model size due to quantization reduces memory consumption during deployment. By reducing the number of memory accesses, inference speed during deployment is improved.

[0095] 3. At low precision, it can basically maintain the model's capabilities in language modeling, learning and understanding contexts with a small number of samples, achieving performance comparable to high-precision models. This avoids the drawback of requiring a large amount of manpower for adjustments after fine-tuning, as is the case with pruning methods.

[0096] Exemplary device

[0097] Figure 7 This is a schematic diagram of the structure of a large model inference device based on quantization-sensory fine-tuning provided in an exemplary embodiment of the present invention. Figure 7 As shown, the device 700 includes:

[0098] The structural modification module 710 is used to modify the original parameter matrix of the large model and determine the horizontal vectorized vector, vertical vectorized vector and low-bit fixed matrix corresponding to the original parameter matrix of each channel.

[0099] The pre-training module 720 is used to perform layer-by-layer parameter quantization fine-tuning pre-training on a large model based on horizontal vectorized vectors, vertical vectorized vectors, and low-bit fixed matrices, and to obtain the horizontal vectorized vector values ​​and vertical vectorized vector values ​​of each channel of the large model.

[0100] The determination module 730 is used to determine the deployment parameter matrix for deployment inference of the large model based on the horizontal vectorized vector values, vertical vectorized vector values ​​and low-bit fixed matrix of each channel of the large model.

[0101] The inference module 740 is used to perform inference analysis on the input data using a large model with a deployment parameter matrix to obtain the inference results of the input data.

[0102] Optionally, the structural modification module 710 includes:

[0103] The decomposition submodule is used to perform horizontal and vertical decomposition on the high-precision original parameter matrix W of each channel of the large model, resulting in a high-precision horizontal vectorized vector T0, a high-precision vertical vectorized vector S0, and a low-bit fixed matrix W0', where the original parameter matrix W of each channel is approximately S0 * W0'.

[0104] Optionally, the bit width of the parameter quantization fine-tuning pre-training is 4.

[0105] Optionally, the pre-trained module 720 includes:

[0106] The first pre-training submodule is used to perform quantization fine-tuning pre-training on the first-layer channels of a large model based on the vertical vectorization vector of the first-layer channel and the low-bit fixed matrix, and to obtain the first-layer quantization result.

[0107] The second pre-training submodule is used to perform quantization fine-tuning pre-training on the next layer channel of the large model based on the vertical vectorization vector and low-bit fixed matrix of the next layer channel, according to the first layer quantization result and the horizontal vectorization vector, to obtain the next quantization result.

[0108] The three pre-training submodules are used to iteratively perform quantization fine-tuning pre-training on the next layer channel by using the quantization results of the previous layer channel of the large model, until the last layer channel of the large model, to obtain the horizontal vectorized vector value and the vertical vectorized vector value of each channel of the large model.

[0109] Optionally, the quantization expression for fine-tuning the pre-trained current layer channels of a large model is:

[0110]

[0111] The quantization expression for the next layer channel of the large model's quantization fine-tuning pre-training is as follows:

[0112] W″0=(T0+ΔT)*W

[0113] In the formula, W is the quantization result of the current channel; W″0 is the quantization result of the next channel of the current channel; S0 is the vertical quantization vector; T0 is the horizontal quantization vector; ΔS represents the gradient update of S0 obtained through fine-tuning training; ΔT represents the gradient update of T0 obtained through fine-tuning training; z0 represents an adjustment parameter; and clamp represents a function that restricts the value to [a, b].

[0114] Exemplary electronic devices

[0115] Figure 8 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. For example... Figure 8 As shown, the electronic device 80 includes one or more processors 81 and memory 82.

[0116] The processor 81 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0117] The memory 82 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 81 may execute the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above, and / or other desired functions. In one example, the electronic device may also include an input device 83 and an output device 84, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0118] In addition, the input device 83 may also include, for example, a keyboard, a mouse, etc.

[0119] The output device 84 can output various information to the outside. The output device 84 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0120] Of course, for the sake of simplicity, Figure 8 Only some of the components of this electronic device relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0121] Exemplary computer program products and computer-readable storage media

[0122] In addition to the methods and apparatus described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0123] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of the present invention. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0124] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0125] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0126] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0128] The block diagrams of devices, systems, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, systems, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0129] The methods and systems of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.

[0130] It should also be noted that in the systems, apparatus, and methods of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered equivalents of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0131] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A large-model inference method based on quantization-perception fine-tuning, characterized in that, include: The original parameter matrix of the large model is structurally modified to determine the horizontal vectorized vector, vertical vectorized vector, and low-bit fixed matrix corresponding to the original parameter matrix of each channel. Based on the horizontal vectorized vector, the vertical vectorized vector, and the low-bit fixed matrix, the large model is pre-trained by performing parameter quantization fine-tuning layer by layer to obtain the horizontal vectorized vector values ​​and vertical vectorized vector values ​​of each channel of the large model. Based on the horizontal vectorized vector values, the vertical vectorized vector values, and the low-bit fixed matrix of each channel of the large model, the deployment parameter matrix for deployment inference of the large model is determined; A large model with the aforementioned deployment parameter matrix is ​​used to perform inference analysis on the input data to obtain the inference results of the input data; The original parameter matrix of the large model is structurally modified to determine the horizontal vectorized vector, vertical vectorized vector, and low-bit fixed matrix corresponding to the original parameter matrix of each channel, including: The original parameter matrix W of each channel of the large model is decomposed horizontally and vertically to obtain high-precision horizontal vectorized vectors. High-precision vertical vectorization And the low-bit fixed matrix W0', wherein the original parameter matrix W≈S0*W0' of each channel; Based on the horizontal vectorized vector, the vertical vectorized vector, and the low-bit fixed matrix, the large model is pre-trained layer by layer with parameter quantization fine-tuning to obtain the horizontal and vertical vectorized vector values ​​of each channel of the large model, including: Based on the vertical vectorization vector and the low-bit fixed matrix of the first-layer channel, the first-layer channel of the large model is pre-trained for quantization fine-tuning to obtain the first-layer quantization result; Based on the vertical vectorization vector of the next layer channel and the low-bit fixed matrix, the next layer channel of the large model is pre-trained for quantization fine-tuning according to the first layer quantization result and the horizontal vectorization vector to obtain the next layer quantization result. The quantization results of the previous layer channel of the large model are used to perform quantization fine-tuning pre-training on the next layer channel until the last layer channel of the large model is reached, so as to obtain the horizontal vectorized vector value and the vertical vectorized vector value of each channel of the large model. The quantization expression for the current channel of the large model quantization fine-tuning pre-training layer is as follows: The quantization expression for the next layer channel of the large model quantization fine-tuning pre-training is as follows: In the formula, W This is the quantization result for this layer's channel; This is the quantization result of the next layer channel in this layer; This is the vertical quantization vector; ΔS represents the gradient update of S0 obtained through fine-tuning training; ΔT represents the gradient update of T0 obtained through fine-tuning training; z0 represents an adjustment parameter.

2. The method according to claim 1, characterized in that, The bit width of the parameter quantization fine-tuning pre-training is 4.

3. A large model inference device based on quantization-sensory fine-tuning, used to implement the method of claim 1, characterized in that, include: The structural modification module is used to modify the original parameter matrix of the large model and determine the horizontal vectorized vector, vertical vectorized vector, and low-bit fixed matrix corresponding to the original parameter matrix of each channel. The pre-training module is used to perform layer-by-layer parameter quantization fine-tuning pre-training on the large model based on the horizontal vectorized vector, the vertical vectorized vector, and the low-bit fixed matrix, and to obtain the horizontal vectorized vector values ​​and vertical vectorized vector values ​​of each channel of the large model. The determination module is used to determine the deployment parameter matrix for deployment inference of the large model based on the horizontal vectorized vector values, the vertical vectorized vector values ​​and the low-bit fixed matrix of each channel of the large model. The inference module is used to perform inference analysis on the input data using a large model with the deployment parameter matrix to obtain the inference results of the input data.

4. The apparatus according to claim 3, characterized in that, The bit width of the parameter quantization fine-tuning pre-training is 4.

5. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-2.

6. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-2.

Images