Large model reasoning acceleration method and device and medium

Abstract

The invention discloses a large model reasoning acceleration method and device and a medium, and the method comprises the steps: constructing a training set and a test set for training a large model according to scene data; expanding a prediction head in the large model to obtain an expanded target large model, and freezing original prediction head parameters in the large model so as to update the prediction head parameters corresponding to the expanded prediction head in the training process of the target large model; training and evaluating the target large model through the training set and the test set to obtain a trained target large model; constructing a calibration data set formed by partial scene data, and inputting the calibration data set into the target large model for forward reasoning to obtain a calibration matrix corresponding to each linear layer; and determining an input matrix and an output matrix corresponding to the linear layer according to the calibration matrix, and quantizing a weight corresponding to the linear layer based on the input matrix to obtain a quantized model weight.

Description

Technical Field

[0001] The present application relates to the field of large language model reasoning optimization, and specifically to a large model reasoning acceleration method, device and medium. Background Art

[0002] Large Language Models (LLMs) have become an important breakthrough in the field of artificial intelligence. They use deep learning technology to learn the grammar, semantics and rules of language in massive text data, and can generate natural and fluent text. With the continuous development of large language models, local large models have attracted widespread attention due to their significant advantages, such as data privacy protection: deploying large models locally can effectively protect users' sensitive data and privacy. By running the model on the user's device or local server, the data does not need to be sent to the cloud for processing, thereby reducing the risk of data leakage. This is particularly important for applications involving personal information, trade secrets or other sensitive data; low latency and high availability: local reasoning does not need to rely on network connection, which can significantly reduce the delay of data transmission, thereby providing instant response speed. In addition, the high availability of local models ensures that the system can still operate normally when there is no network connection or cloud service is unavailable; model fine-tuning and weight ownership: in the local environment, users can directly fine-tune the model to meet specific needs and have full control over the weights of the model. This is particularly critical for enterprises or research institutions that need customized solutions. At the same time, having control over the weights also avoids the uncertainty and potential risks that may arise from relying on third-party cloud service providers.

[0003] Although local large models have many advantages, their reasoning process often requires high computing resources. In a local environment that lacks sufficient computing resources, reasoning speed may become a bottleneck, affecting user experience. Summary of the invention

[0004] In order to solve the above problems, this application proposes a large model reasoning acceleration method, including: Based on the model optimization method of the large model, the scene data corresponding to the large model is collected, and according to the scene data, a training set and a test set for training the large model are constructed; Expanding the prediction head in the large model to obtain an expanded target large model, and freezing the original prediction head parameters in the large model, so that during the training process of the target large model, the prediction head parameters corresponding to the expanded prediction head are updated; wherein the multiple prediction heads can realize parallel decoding between tokens, and the prediction heads are represented as linear layers; The target large model is trained and evaluated by the training set and the test set respectively to obtain a trained target large model; Constructing a calibration data set consisting of partial scene data, inputting the calibration data set into the target large model for forward reasoning, and obtaining a calibration matrix corresponding to each linear layer; wherein the calibration matrix is ​​composed of activation values ​​corresponding to the linear layer; According to the calibration matrix, the input matrix and the output matrix corresponding to the linear layer are determined, and the weights corresponding to the linear layer are quantized based on the input matrix to obtain the quantized model weights.

[0005] In one implementation of the present application, the weights corresponding to the linear layer are quantized based on the input matrix to obtain the quantized model weights, specifically including: Determine the average value between each row of input data in the input matrix, and use the average value as the importance of each column weight in the weight matrix corresponding to the linear layer; For each linear layer, according to the importance, determine a set of scaling coefficients corresponding to the weight matrix, scale the weights of the linear layer according to the set of scaling coefficients, and perform an inverse operation on a previous linear layer of the linear layer; The linear layer is quantized by a preset MinMax algorithm to obtain a quantized model weight.

[0006] In one implementation of the present application, determining a set of scaling coefficients corresponding to the weight matrix according to the importance specifically includes: Based on a greedy search algorithm, a scaling factor corresponding to each column weight is determined according to the importance, so as to scale the weight and the input data in the input matrix according to the scaling factor, so that the mean square error between the weight and the input data obtained after scaling and the input matrix is ​​minimized; wherein the scaling factor is obtained by multiplying the importance and a specified value within a preset value range; According to the scaling factor corresponding to each column of weight, a scaling factor set corresponding to the weight matrix is ​​constructed.

[0007] In one implementation of the present application, the linear layer is quantized by a preset MinMax algorithm to obtain a quantized model weight, which specifically includes: Based on different quantization methods, respectively determine the scaling factors corresponding to the linear layers; The linear layer is quantized according to the scaling factor by the preset MinMax algorithm to obtain the quantized model weight. The specific calculation method is:

[0008] in, and They represent the vector of each column weight before quantization and after quantization, s is the scaling factor, and z is the zero point value.

[0009] In one implementation of the present application, based on different quantization methods, scaling factors corresponding to the linear layers are determined respectively, specifically including: The quantization method includes symmetric quantization and asymmetric quantization; When the quantization mode is the symmetric quantization, the scaling factor corresponding to the linear layer is determined by the following formula:

[0010] When the quantization mode is the asymmetric quantization, the scaling factor corresponding to the linear layer is determined by the following formula:

[0011] Where N represents the number of quantization bits.

[0012] In one implementation of the present application, the target large model is trained and evaluated by the training set and the test set, respectively, specifically including: The target large model is trained by using the training set, a loss value corresponding to the target large model is calculated during the training process, and a prediction head parameter corresponding to the extended prediction head is updated by using a preset optimization algorithm to minimize the loss value; The above process is repeated until the target large model reaches convergence; wherein the loss function used to calculate the loss value is expressed as:

[0013] Where k is the number of extended prediction heads, represents the prediction probability of the token at the corresponding position of the k-th prediction head, is a constant used to balance the losses of each prediction head.

[0014] In one implementation of the present application, the prediction head in the large model is expanded to obtain an expanded target large model, specifically including: After the last hidden layer of the large model, a plurality of newly added prediction heads are added; wherein the newly added prediction heads include a first linear layer, a residual structure composed of an activation layer, and a second linear layer; Initialize the first linear layer to an all-zero matrix, and initialize the second linear layer so that the weights of the second linear layer are consistent with the original prediction head; Based on the initialized new prediction head, the expanded large model is obtained.

[0015] In one implementation of the present application, based on a model optimization method of a large model, scene data corresponding to the large model is collected, specifically including: When the model optimization method is supervised fine-tuning, collecting the data set used in the supervised fine-tuning stage as the scene data corresponding to the large model; When the model optimization method is reinforcement learning, the feature data generated by the large model through the distillation technology is used as the scene data corresponding to the large model.

[0016] The present application embodiment provides a large model reasoning acceleration device, the device comprising: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above large model reasoning acceleration methods described.

[0017] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows: A large model reasoning acceleration method as described in any of the above items.

[0018] The large model reasoning acceleration method proposed in this application can bring the following beneficial effects: By expanding the prediction head of the large model and realizing parallel decoding between tokens, the parallel processing capability of the model when processing text or sequence data is significantly improved, which effectively reduces the waiting time during the reasoning process, allowing the model to complete the processing of large amounts of data in a shorter time, thereby improving the overall reasoning speed.

[0019] During the training process, by freezing the original prediction head parameters in the large model, only the extended prediction head parameters are updated, which reduces the number of parameters that need to be optimized during the training process, reduces the computational complexity and memory usage. At the same time, for limited local computing resources, this optimization method enables the large model to use these resources more effectively, improving resource utilization efficiency.

[0020] Quantizing the linear layer weights through the calibration matrix reduces the storage space of the model parameters and may further improve the inference speed through hardware acceleration. At the same time, the calibration process ensures that the quantized model can maintain good performance and reduce the accuracy loss introduced by quantization. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a large model reasoning acceleration method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a large model reasoning acceleration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0023] The embodiment of the present application proposes a method for accelerating large model reasoning, which combines parallel decoding and large model quantization technology that adjusts the large model output head structure, effectively improving the reasoning speed, thereby achieving efficient large model reasoning under limited local computing resources.

[0024] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0025] like Figure 1 As shown, a large model reasoning acceleration method provided by an embodiment of the present application includes: S101: Based on the model optimization method of the large model, the scene data corresponding to the large model is collected, and according to the scene data, a training set and a test set for training the large model are constructed.

[0026] After selecting the large model, collect scenario data that meets the model usage scenario for the large model. In this process, it is necessary to determine it according to the model optimization method of the large model. Model optimization methods are divided into two categories, namely supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). If it is only a large model that has been fine-tuned by supervision, collect the data set used in the supervised fine-tuning stage as the scenario data corresponding to the large model; if the large model has undergone the RLHF process or has no training data, then the feature data generated by the large model through distillation technology is needed as the scenario data of the large model. Specifically, the relevant text in the actual usage scenario is input to the large model, and then the reply text of the large model is obtained, and then multiple sets of input and output are collected as training data for the large model that will be fine-tuned and changed later. After obtaining the scenario data, construct the corresponding training set and test set according to the preset ratio.

[0027] In one embodiment, the embodiment of the present application may select the open source model Qwen1.5-14B-Chat as the large model, which has 14 billion parameters and supports a context length of up to 32,768 tokens, and is suitable for tasks such as text generation, translation, summarization, and question-answering systems. Since the model has undergone a reinforcement learning with human feedback (RLHF) process, the large model is selected to generate scene data that conforms to its own output characteristics through distillation technology. Download the open source LCCC-base Chinese dialogue data set, use the TGI framework to start the service of the model, send the preprocessed dialogue data to the model in the TGI service in batches, and get the corresponding reply. After processing, the new dialogue data pairs are saved as training data for the large model that will be fine-tuned and changed later. Divide the data into training set and test set in a ratio of 8:2.

[0028] S102: Expand the prediction head in the large model to obtain the expanded target large model, and freeze the original prediction head parameters in the large model so that during the training process of the target large model, the prediction head parameters corresponding to the expanded prediction head can be updated; wherein, multiple prediction heads can realize parallel decoding between tokens, and the prediction head is represented as a linear layer.

[0029] There is only one original prediction head behind the last hidden layer of the original large model (essentially a linear layer of size D x V, where D is the hidden layer dimension and V is the number of words), which is responsible for predicting the next word token to be output by the large model. In order to improve the reasoning speed of the large model, the structure of the original large model needs to be improved accordingly so that it can achieve parallel decoding between different word tokens. Based on the expanded target large model, the original prediction head parameters corresponding to each prediction head in the original model are frozen, so that during the training process of the target large model, only the newly added prediction head parameters are updated. Freezing the original prediction head parameters means that these parameters do not need to be updated during the training process, thereby reducing the total number of parameters that need to be optimized, which greatly reduces the computational complexity of training, shortens the training time, and makes the model training process more efficient.

[0030] Specifically, several new prediction heads are added after the last hidden layer of the large model. The new prediction heads include a first linear layer, a residual architecture composed of a SiLu activation layer, and a second linear layer. The size of the first linear layer is D x D, and the size of the second linear layer is D x V. The first linear layer is initialized to an all-0 matrix, and the second linear layer is initialized so that the weights of the second linear layer are consistent with the original prediction heads. After the initialization operation is completed, the expanded large model can be obtained based on the initialized new prediction heads. For example, k new prediction heads are introduced after the last hidden layer of the model, k is 3, and the expanded prediction heads predict what the second, third, and fourth tokens below are respectively.

[0031] S103: The target large model is trained and evaluated through the training set and the test set respectively to obtain a trained target large model.

[0032] After completing the expansion of the large model, the expanded target large model needs to be retrained so that the newly added prediction head can be effectively integrated with the original model part, so that the expanded model can work in an overall coordinated manner.

[0033] Specifically, since each training round only updates the prediction head parameters corresponding to the expanded prediction head, when training the model, taking k as 3 as an example, the input is the decoded tokens of one or more paragraphs of text, and the output label should point to the second to fourth tokens after the corresponding text. The loss function uses the cross entropy loss function L between the predicted value and the true value, and its formula is:

[0034] Where k is the number of extended prediction heads, represents the prediction probability of the token at the corresponding position of the k-th prediction head, is a constant used to balance the losses of each prediction head.

[0035] During the training process, the above loss function is used to cyclically use a batch of samples in the training set to input the model to calculate the loss value, and then the gradient is calculated through the back propagation algorithm, and the preset optimization algorithm (such as Adam optimizer) is used to update the prediction head parameters corresponding to each expanded prediction head to minimize the loss value. Repeat the above process until there is little change in multiple consecutive training rounds, indicating that the target large model has reached convergence, and the target large model needs to be evaluated through the test data set to confirm that its accuracy reaches the expected target. At this point, the trained target large model is obtained.

[0036] S104: construct a calibration data set consisting of partial scene data, input the calibration data set into the target large model for forward reasoning, and obtain a calibration matrix corresponding to each linear layer; wherein the calibration matrix is ​​composed of activation values ​​corresponding to the linear layer.

[0037] Model quantization refers to using fewer bits to represent model parameters, thereby reducing the size of the model and speeding up the inference process. The target large model completed by the above training is 16 bits, that is, a weight parameter in the model is represented by a 16-bit value. In order to increase the inference speed, it needs to be compressed to 8 bits with almost no accuracy loss. This process needs to be implemented based on the calibration matrix, which is obtained by forward reasoning on a calibration data set composed of partial scene data, and is specifically composed of activation values ​​corresponding to the linear layer.

[0038] Therefore, the calibration matrix is ​​obtained by inputting the calibration dataset into the target large model to save the activation values ​​of each linear layer. Specifically, a hook function is defined in the code to capture the activation values ​​before and after all linear layers in the loaded target large model. Then the calibration dataset is used to perform forward inference on the model, and the hook function automatically saves the activation values ​​of the linear layer input into a set of calibration matrices X for use in subsequent steps. For the first layer of the model, X contains the input embedding from the calibration dataset, and for the middle layer, X contains the input activation of the current layer, that is, the output activation of the previous layer.

[0039] S105: According to the calibration matrix, the input matrix and the output matrix corresponding to the linear layer are determined, and the weights corresponding to the linear layer are quantized based on the input matrix to obtain the quantized model weights.

[0040] Based on the calibration matrix obtained above, the input matrix and output matrix corresponding to each linear layer are determined, and then the weights of the linear layer are quantized based on the input matrix, and the model weights are compressed from 16 bits to 8 bits to obtain the quantized model weights.

[0041] Before quantizing the model weights, a series of linear layer scaling operations are required to reduce the difficulty of subsequent quantization. First, determine the average value between each row of input data in the input matrix, and use the average value as the importance of each column weight in the weight matrix corresponding to the linear layer. For each linear layer, determine the set of scaling factors corresponding to the weight matrix according to the importance, and the set of scaling factors reflects the difference in importance of different column weights.

[0042] Among them, the set of scaling coefficients needs to be obtained by greedy search, aiming to minimize the mean square error between the scaled weights and input data and the original input matrix by adjusting the scaling coefficient of each column of weights. First, the scaling coefficient corresponding to each column of weights is determined according to the importance, and the weights and input data in the input matrix are scaled according to the scaling coefficient. This process is to multiply each column of weights by the scaling coefficient to obtain a new weight, and multiply the input data by the scaling coefficient to obtain new input data, and then minimize the mean square error between the scaled weights and input data and the input matrix. Among them, the scaling coefficient is obtained by multiplying the importance and the specified value within the preset numerical range. The preset numerical range here is [0,1]. That is to say, the scaling coefficient is obtained by multiplying the importance and a value in [0,1]. According to the scaling coefficient corresponding to each column of weights obtained by the above process, a set of scaling coefficients corresponding to the weight matrix is ​​constructed.

[0043] By applying a scaling factor, the weight matrix can be scaled to emphasize important features and potentially weaken the influence of unimportant features. In addition, since the scaling operation changes the distribution of the weight matrix, the previous layer of the linear layer also needs to be inverted to ensure that the overall output of the model is not affected.

[0044] Quantization is the process of converting the floating-point weights in the model to fixed-point numbers or lower-precision floating-point numbers, which aims to reduce the size of the model and improve the inference speed. The MinMax algorithm is a simple quantization method that determines the quantization parameters by finding the minimum and maximum values ​​in the weight matrix. Therefore, the MinMax algorithm can be used to quantize the linear layer and obtain the quantized model weights.

[0045] Specifically, based on different quantization methods, scaling factors corresponding to the linear layers are determined respectively. The quantization methods include symmetric quantization and asymmetric quantization.

[0046] When the quantization method is symmetric quantization, the scaling factor corresponding to the linear layer is:

[0047] When the quantization method is asymmetric quantization, the scaling factor corresponding to the linear layer is

[0048] Where N represents the number of quantization bits.

[0049] Then, the preset MinMax algorithm is used to quantize the linear layer according to the above scaling factor to obtain the quantized model weights. The specific calculation method is:

[0050] in, and They represent the vector of each column weight before quantization and after quantization, s is the scaling factor, and z is the zero point value.

[0051] After quantizing the target large model, use the validation data set to evaluate the accuracy and performance of the quantized model and confirm that it is within an acceptable range. After verification and evaluation, deploy it to the actual application environment and continue to monitor the performance of the target large model to ensure that the quantized model still performs well in real scenarios. At the same time, according to application requirements, the quantization parameters of the model (such as the number of quantization bits N) can be further adjusted to optimize inference speed and resource usage.

[0052] The above are embodiments of the method proposed in this application. Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0053] Figure 2 This is a schematic diagram of the structure of a large model reasoning acceleration device provided in an embodiment of the present application. Figure 2 As shown, including: at least one processor; and, at least one processor is communicatively connected to a memory; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute a large model reasoning acceleration method as described in any of the above items.

[0054] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows: A large model reasoning acceleration method as described in any of the above items.

[0055] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0056] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0057] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0058] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0059] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0060] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0061] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0062] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0063] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0064] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0065] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A large model reasoning acceleration method, characterized in that: The method comprises: Based on the model optimization method of the large model, the scene data corresponding to the large model is collected, and according to the scene data, a training set and a test set for training the large model are constructed; Expanding the prediction head in the large model to obtain an expanded target large model, and freezing the original prediction head parameters in the large model, so that during the training process of the target large model, the prediction head parameters corresponding to the expanded prediction head are updated; wherein the multiple prediction heads can realize parallel decoding between tokens, and the prediction heads are represented as linear layers; The target large model is trained and evaluated by the training set and the test set respectively to obtain a trained target large model; Constructing a calibration data set consisting of partial scene data, inputting the calibration data set into the target large model for forward reasoning, and obtaining a calibration matrix corresponding to each linear layer; wherein the calibration matrix is ​​composed of activation values ​​corresponding to the linear layer; According to the calibration matrix, the input matrix and the output matrix corresponding to the linear layer are determined, and the weights corresponding to the linear layer are quantized based on the input matrix to obtain the quantized model weights.

2. A large model reasoning acceleration method according to claim 1, characterized in that: The weights corresponding to the linear layer are quantized based on the input matrix to obtain the quantized model weights, specifically including: Determine the average value between each row of input data in the input matrix, and use the average value as the importance of each column weight in the weight matrix corresponding to the linear layer; For each linear layer, according to the importance, determine a set of scaling coefficients corresponding to the weight matrix, scale the weights of the linear layer according to the set of scaling coefficients, and perform an inverse operation on a previous linear layer of the linear layer; The linear layer is quantized by a preset MinMax algorithm to obtain a quantized model weight.

3. A large model reasoning acceleration method according to claim 2, characterized in that: Determining a set of scaling coefficients corresponding to the weight matrix according to the importance specifically includes: Based on a greedy search algorithm, a scaling factor corresponding to each column weight is determined according to the importance, so as to scale the weight and the input data in the input matrix according to the scaling factor, so that the mean square error between the weight and the input data obtained after scaling and the input matrix is ​​minimized; wherein the scaling factor is obtained by multiplying the importance and a specified value within a preset value range; According to the scaling factor corresponding to each column of weight, a scaling factor set corresponding to the weight matrix is ​​constructed.

4. A large model reasoning acceleration method according to claim 2, characterized in that: The linear layer is quantized by a preset MinMax algorithm to obtain a quantized model weight, which specifically includes: Based on different quantization methods, respectively determine the scaling factors corresponding to the linear layers; The linear layer is quantized according to the scaling factor by the preset MinMax algorithm to obtain the quantized model weight. The specific calculation method is: in, and They represent the vector of each column weight before quantization and after quantization, s is the scaling factor, and z is the zero point value.

5. A large model reasoning acceleration method according to claim 4, characterized in that: Based on different quantization methods, the scaling factors corresponding to the linear layer are determined respectively, specifically including: The quantization method includes symmetric quantization and asymmetric quantization; When the quantization mode is the symmetric quantization, the scaling factor corresponding to the linear layer is determined by the following formula: When the quantization mode is the asymmetric quantization, the scaling factor corresponding to the linear layer is determined by the following formula: Where N represents the number of quantization bits.

6. A large model reasoning acceleration method according to claim 1, characterized in that: The target large model is trained and evaluated by the training set and the test set respectively, specifically including: The target large model is trained by using the training set, a loss value corresponding to the target large model is calculated during the training process, and a prediction head parameter corresponding to the extended prediction head is updated by using a preset optimization algorithm to minimize the loss value; The above process is repeated until the target large model reaches convergence; wherein the loss function used to calculate the loss value is expressed as: Where k is the number of extended prediction heads, represents the prediction probability of the token at the corresponding position of the k-th prediction head, is a constant used to balance the losses of each prediction head.

7. A large model reasoning acceleration method according to claim 1, characterized in that: Expanding the prediction head in the large model to obtain an expanded target large model specifically includes: After the last hidden layer of the large model, a plurality of newly added prediction heads are added; wherein the newly added prediction heads include a first linear layer, a residual structure composed of an activation layer, and a second linear layer; Initialize the first linear layer to an all-zero matrix, and initialize the second linear layer so that the weights of the second linear layer are consistent with the original prediction head; Based on the initialized new prediction head, the expanded large model is obtained.

8. A large model reasoning acceleration method according to claim 1, characterized in that: Based on the model optimization method of the large model, the scene data corresponding to the large model is collected, specifically including: When the model optimization method is supervised fine-tuning, collecting the data set used in the supervised fine-tuning stage as the scene data corresponding to the large model; When the model optimization method is reinforcement learning, the feature data generated by the large model through the distillation technology is used as the scene data corresponding to the large model.

9. A large model reasoning acceleration device, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a large model reasoning acceleration method as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: A large model reasoning acceleration method as described in any one of claims 1 to 8.

Images