A method and system for lossless acceleration of large language model reasoning through model pruning

Abstract

The present invention proposes a method and system for losslessly accelerating the reasoning of a large language model by model pruning, which belongs to the field of artificial intelligence technology. The method and system include: using a genetic algorithm to select skip layers to perform model pruning on an LLM, accelerating by skipping certain layers in the LLM during the reasoning process of accelerating the large language model, and generating a speculated Token sequence; then, in the verification phase, the original LLM is used to perform a one-time forward pass to verify these speculated Token sequences, and the final output Token sequence is composed of the verified Tokens, ensuring that the final output is consistent with the output generated by the unchanged LLM, thereby achieving reasoning acceleration while maintaining the output quality.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a method and system for performing model pruning losslessly to accelerate large language model reasoning. Background Art

[0002] Large language models (LLMs), such as GPT-3 / 4, PaLM, and LLaMA, have achieved excellent performance in various natural language tasks, including text summarization, language translation, question answering, etc. These models are trained with deep learning and large amounts of data to generate accurate and coherent text output. However, due to the large size of the models, their inference speed is slow and the computational cost is expensive, which becomes a significant bottleneck in scenarios with limited resources or high real-time requirements.

[0003] In order to solve the problem of inference efficiency of large language models, researchers have proposed a variety of methods:

[0004] Reference text decoding-based method: This method builds a text vector knowledge base and retrieves text fragments similar to the input prompt during the autoregressive decoding process. Then, a parallel decoding and verification strategy is used to accelerate the reasoning process of large language models. Although this method performs well in retrieval-enhanced generation scenarios, its applicability is limited.

[0005] Method based on inference decoding: This method builds an efficient inference model, first uses the model to generate inference tokens through autoregressive decoding, then inputs these tokens into the original model for parallel reasoning and verification to ensure the consistency of the generated results with the original model. This method expands the applicable generation scenarios, but the introduction of additional inference models leads to an increase in memory usage, which is difficult to deploy in an environment with limited hardware resources. Summary of the invention

[0006] The present invention aims at solving the problems in the prior art, and the technical solution adopted in this application is as follows:

[0007] In a first aspect, the present invention provides a method for performing model pruning to losslessly accelerate large language model reasoning, comprising: obtaining a large language model to be accelerated, and performing model pruning acceleration by skipping certain layers in the LLM during the process of accelerating the reasoning of the large language model. And performing lossless reasoning in combination with speculative decoding technology, thereby realizing overall LLM lossless reasoning acceleration.

[0008] Further, determining to skip certain layers in the LLM step includes:

[0009] Use genetic algorithm to select skip layers and generate inferred token sequences;

[0010] The inferred Token sequence is verified using inferred decoding, and the verified Tokens constitute the final output Token sequence.

[0011] Furthermore, the specific steps of selecting the skip layer by using the genetic algorithm include:

[0012] Problem definition: The problem of choosing which layers to skip in a large language model is defined as an optimization problem, the goal is to minimize the average inference time of each token;

[0013] Parameter encoding: Encode the decision of whether to skip each layer into a binary value, 0 means not to skip, 1 means to skip, forming a binary string representing an individual;

[0014] Initial population generation: randomly generate a group of initial individuals to form an initial population, each of which represents a different layer skipping combination;

[0015] Fitness evaluation: Evaluate each individual, apply its layer skipping combination to each individual, and measure the average inference time of each token of the model to obtain the fitness value;

[0016] Iterative evolution: repeat the selection operation, crossover operation, and mutation operation steps until the termination condition is met to obtain the final population; selection operation: select excellent individuals according to the fitness value to enter the next generation; crossover operation: exchange information between individuals through crossover operation to generate new layer skipping combinations; mutation operation: randomly change certain genes of individuals with a certain probability;

[0017] Determine the optimal solution: Select the layer skipping combination with the shortest average inference time from the final population as the optimal layer skipping combination.

[0018] Furthermore, the specific steps of verifying the inferred Token sequence by inferred decoding include:

[0019] Inference stage: Use the optimal layer skipping combination determined by the genetic algorithm to selectively skip this part of the intermediate layers and generate an inferred Token sequence;

[0020] Verification phase: The Token sequence generated in the inference phase is used as the original model input, and a one-time forward pass is performed to obtain the prediction result of the inferred Token sequence; if the prediction result of the inferred Token sequence is consistent with the prediction result of the original model's predicted Token, the inferred Token sequence is accepted; otherwise, the inferred Token sequence is rejected and the predicted Token of the original model is used;

[0021] Generate the final output: The verified and accepted Token sequence will be spliced ​​into the output sequence; for the rejected Token position, the predicted Token of the original model is directly spliced ​​into the output sequence to obtain the final output sequence.

[0022] In a second aspect, a system for performing model pruning and lossless acceleration of large language model reasoning includes:

[0023] The processing module is configured to: obtain a large language model to be accelerated, perform model pruning acceleration by skipping certain layers in the LLM during the process of accelerating the reasoning of the large language model, and realize lossless reasoning in combination with the speculative decoding strategy, thereby realizing lossless accelerated reasoning as a whole; the steps include:

[0024] Use genetic algorithm to select skip layers and generate inferred token sequences;

[0025] The inferred Token sequence is verified using inferred decoding, and the verified Tokens constitute the final output Token sequence.

[0026] In a third aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute a method for performing model pruning losslessly to accelerate large language model reasoning as described in any one of the first aspects.

[0027] In a fourth aspect, the present invention provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute a method for losslessly accelerating large language model reasoning by model pruning as described in any one of the first aspects.

[0028] Compared with the prior art, the advantages and positive effects of the present invention are:

[0029] 1. This paper proposes a novel method for losslessly accelerating the reasoning of large language models by model pruning based on genetic algorithms, aiming to accelerate the reasoning process of large language models without the need for additional auxiliary models or increased memory usage. This method uses genetic algorithms to select skipped layers to perform model pruning on LLMs, selectively skips certain layers in the LLMs in the inference phase, quickly generates inferred tokens, and then uses the original LLMs for a one-time forward pass in the verification phase to verify these inferred outputs, ensuring that the final output is consistent with the output generated by the unchanged LLM, thereby achieving inference acceleration while maintaining output quality.

[0030] 2. The present invention does not require an additional inference model: Compared with other methods that require the selection or training of a new inference model, this solution directly performs layer selection based on the original model without adding additional memory burden. Plug-and-play acceleration solution: This method does not require any additional training or adjustment of the model and can be quickly integrated into existing systems. Maintain output quality: By intelligently selecting layer skipping combinations, while achieving reasoning acceleration, ensure that the output results are consistent with the original model and guarantee the generation quality. Wide applicability: It is suitable for various types of large language models and different natural language processing tasks, and has broad application potential.

[0031] 3. This solution combines genetic algorithms for effective interaction of model pruning and speculative decoding, achieving lossless acceleration of the reasoning process of large language models, and providing an efficient solution for natural language processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 It is a schematic diagram of the genetic algorithm model pruning process of the present invention;

[0034] Figure 2 This is a schematic diagram of accelerating large language model reasoning by model pruning according to the present invention. DETAILED DESCRIPTION

[0035] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0037] Embodiment 1, as Figure 1-Figure 2 As shown, this embodiment provides a method for losslessly accelerating the reasoning of a large language model by model pruning based on a genetic algorithm. The method of this embodiment intelligently and selectively skips certain layers in the large language model and combines a speculative decoding strategy to losslessly accelerate the reasoning process. This method does not require additional neural network training or memory usage, and provides a plug-and-play acceleration solution. Figure 1 As shown, the specific implementation is as follows:

[0038] Step S1, model selection: The acceleration scheme of this embodiment is designed to achieve seamless integration and is applicable to various existing large language models. It does not require additional network training or memory usage and is a plug-and-play solution. To verify the effectiveness of this scheme, the open source LLaMA2-70b model is selected as the original model for experiment. The LLaMA2-70b model is used in a variety of natural language processing tasks due to its powerful performance and wide application.

[0039] By applying the method of the present invention to the original model, this embodiment can directly accelerate the reasoning process without introducing additional auxiliary models or occupying more hardware resources.

[0040] Step S2, skip layer selection: Skip layer selection is the core part of this solution, which involves solving an optimization problem. In the process of speculative decoding, skipping more layers can speed up the inference speed, but it may also reduce the acceptance rate of tokens in the verification phase. If too many key layers are skipped, the inferred tokens may be inconsistent with the output of the original model, resulting in an increase in the rejection rate in the verification phase, thereby prolonging the overall inference time.

[0041] Therefore, the goal of skip layer selection is to minimize the average inference time of each token, which includes the generation time of the inference phase and the correction time of the verification phase. To this end, this embodiment applies a genetic algorithm to intelligently select the best layer skip combination. This solution is experimented on the CNN / DM public dataset, and the final skip layer combination selected by the genetic algorithm can achieve a 1.5-fold performance improvement.

[0042] The specific steps are as follows:

[0043] Step S11, problem definition: The problem of skipping layers is defined as an optimization problem, and the goal is to minimize the average inference time of each token by skipping appropriate layers. It is necessary to balance acceleration and accuracy in order to find a layer combination that can speed up inference while maintaining a high verification rate.

[0044] Step S12, parameter encoding: Encode the skip decision of each layer into a binary value, where 0 means not to skip the layer and 1 means to skip the layer. These binary values ​​are concatenated into an individual, representing a set of layer skip combinations.

[0045] Step S13, initial population generation: randomly generate a group of initial individuals to form an initial population. Each individual represents a different layer skipping strategy, and the diversity of the population helps to fully explore the solution space.

[0046] Step S14, fitness evaluation: evaluate each individual, that is, apply the layer skip combination represented by the individual in the LLM, and then measure and record the inference time after using the combination. Specifically, apply the layer skip combination to each individual, measure the average inference time of each token of the model to obtain the fitness value. The fitness value is defined as the inverse of the inference time, that is, the contribution of the acceleration effect to the fitness.

[0047] Step S15, iterative evolution: repeatedly repeat the selection, crossover and mutation operations, and perform multiple generations of evolution until a predetermined termination condition is reached, such as reaching the maximum number of iterations or the fitness is no longer significantly improved. Selection operation: select excellent individuals according to fitness (i.e., the inverse of reasoning time) to enter the next generation. Specifically, this embodiment uses the tournament selection method to randomly select a group of individuals from the current population, and then select the individuals with the highest fitness from these individuals to enter the next generation, and repeat this process until the size of the new population reaches the required number. Crossover operation: exchange information between individuals through the crossover operation to generate new layer skipping combinations. Specifically, through the crossover operation, two parent individuals are selected to exchange genetic information to generate offspring individuals. This embodiment adopts the method of using multi-point crossover in the early stage of the genetic algorithm iteration and single-point crossover in the later stage of the iteration to create a new layer skipping combination. Mutation operation: randomly change certain genes of an individual with a certain probability to explore a wider solution space. Specifically, this scheme mutates the genes of an individual with a probability of 0.01, that is, randomly changes the skipping state of certain layers. This increases the exploration of the solution space and helps to avoid local optimality.

[0048] Step S16, optimal solution determination: select the individual with the highest fitness from the final population, that is, the layer skipping combination with the shortest average inference time. The layer skipping combination of this individual can achieve the best balance between quickly generating inferred tokens and ensuring a high acceptance rate in the verification phase.

[0049] Step S3, inferring the decoding process, using the optimal layer skipping combination determined by the genetic algorithm to perform the inferring decoding process can achieve inference acceleration. Figure 2 As shown in Figure 1, the process includes interactive reasoning in the inference phase and the verification phase:

[0050] Step S31, inference stage: for each input sequence, the optimal layer skipping combination determined by the genetic algorithm is used to selectively skip this part of the intermediate layers and quickly generate inference Tokens.

[0051] Step S32, verification phase: take the token sequence generated in the inference phase as input and use the original LLaMA2-70b model for a one-time forward pass. The preliminary token sequence output provided by the inference phase may be inconsistent with the final output of the original model. A verification process is required to verify each inferred token to ensure that it is consistent with the output of the original model. If the inferred token is consistent with the prediction of the original model, the token is accepted; otherwise, the token is rejected and the predicted token of the original model is used.

[0052] Step S33, generate the final output: the verified and accepted token sequences will be spliced ​​into the final output sequence. For the rejected token positions, the predicted tokens of the original model are directly used to ensure the integrity and consistency of the output. This process ensures that the inference is accelerated while ensuring that the output results maintain the same generation quality as the original model.

[0053] The solution of this embodiment shows significant advantages in speculative decoding:

[0054] No additional inference model is required: Compared with other methods that require selecting or training a new inference model, this solution directly selects layers based on the original model without adding additional memory burden.

[0055] Plug-and-play acceleration solution: This method does not require any additional training or adjustment of the model and can be quickly integrated into existing systems.

[0056] Maintain output quality: By intelligently selecting layer skip combinations, we can achieve inference acceleration while ensuring that the output results are consistent with the original model and the generation quality is guaranteed.

[0057] Wide applicability: It is applicable to various types of large language models and different natural language processing tasks, and has wide application potential.

[0058] This embodiment combines the genetic algorithm to perform effective interaction of model pruning and speculative decoding, achieves lossless acceleration of the reasoning process of a large language model, and provides an efficient solution for natural language processing tasks.

[0059] Embodiment 2: This embodiment provides a system for performing model pruning losslessly accelerating large language model reasoning, including:

[0060] The processing module is configured to: obtain a large language model to be accelerated, perform model pruning acceleration by skipping certain layers in the LLM during the reasoning process of the large language model, and realize lossless reasoning in combination with the speculative decoding strategy, thereby realizing lossless accelerated reasoning as a whole; the steps include:

[0061] Use genetic algorithm to select skip layers and generate inferred token sequences;

[0062] The inferred Token sequence is verified using inferred decoding, and the verified Tokens constitute the final output Token sequence.

[0063] Embodiment 3, this embodiment provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute a method for performing model pruning losslessly to accelerate large language model reasoning as described in any one of Embodiments 1.

[0064] Embodiment 4. This embodiment provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute a method for performing model pruning losslessly to accelerate large language model reasoning as described in any one of Embodiments 1.

[0065] The electronic device may include: a processor, a memory and a communication unit. These components communicate via one or more buses. Those skilled in the art will appreciate that the structure of the electronic device does not limit the embodiments of the present invention, and it may be a bus structure or a star structure, or a combination of certain components, or different component arrangements.

[0066] The communication unit is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0067] The processor is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0068] The memory is used to store the execution instructions of the processor. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0069] When the execution instructions in the memory are executed by the processor, the electronic device is enabled to execute part or all of the steps of Embodiment 1.

[0070] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for performing model pruning to losslessly accelerate large language model reasoning, characterized in that: include: By skipping some layers in the LLM to accelerate the reasoning of large language models, model pruning is accelerated, and lossless reasoning is achieved in combination with the speculative decoding strategy, so as to achieve lossless accelerated reasoning as a whole, which is suitable for various types of large language models and different natural language processing tasks; the steps include: Use genetic algorithm to select skip layers and generate inferred token sequences; The inferred Token sequence is verified using inferred decoding, and the verified Tokens form the final output Token sequence; The specific steps of selecting the skip layer by using the genetic algorithm include: Problem definition: The problem of choosing which layers to skip in a large language model is defined as an optimization problem, the goal is to minimize the average inference time of each token; Parameter encoding: Encode the decision of whether to skip each layer into a binary value, 0 means not to skip, 1 means to skip, forming a binary string representing an individual; Initial population generation: randomly generate a group of initial individuals to form an initial population, each of which represents a different layer skipping combination; Fitness evaluation: Evaluate each individual, apply its layer skipping combination to each individual, and measure the average inference time of each token of the model to obtain the fitness value; Iterative evolution: repeat the selection operation, crossover operation, and mutation operation steps until the termination condition is met to obtain the final population; selection operation: select excellent individuals according to the fitness value to enter the next generation; crossover operation: exchange information between individuals through crossover operation to generate new layer skipping combinations; mutation operation: randomly change certain genes of individuals with a certain probability; Determine the optimal solution: Select the layer skipping combination with the shortest average inference time from the final population as the optimal layer skipping combination.

2. A system for performing model pruning to losslessly accelerate large language model reasoning, characterized in that: include: The processing module is configured to: perform model pruning acceleration by skipping certain layers in the LLM during the process of accelerating the reasoning of the large language model, and realize lossless reasoning in combination with the speculative decoding strategy, so as to realize lossless accelerated reasoning as a whole, which is applicable to various types of large language models and different natural language processing tasks; the steps include: Use genetic algorithm to select skip layers and generate inferred token sequences; The inferred Token sequence is verified using inferred decoding, and the verified Tokens form the final output Token sequence; The specific steps of selecting the skip layer by using the genetic algorithm include: Problem definition: The problem of choosing which layers to skip in a large language model is defined as an optimization problem, the goal is to minimize the average inference time of each token; Parameter encoding: Encode the decision of whether to skip each layer into a binary value, 0 means not to skip, 1 means to skip, forming a binary string representing an individual; Initial population generation: randomly generate a group of initial individuals to form an initial population, each of which represents a different layer skipping combination; Fitness evaluation: Evaluate each individual, apply its layer skipping combination to each individual, and measure the average inference time of each token of the model to obtain the fitness value; Iterative evolution: repeat the selection operation, crossover operation, and mutation operation steps until the termination condition is met to obtain the final population; selection operation: select excellent individuals according to the fitness value to enter the next generation; crossover operation: exchange information between individuals through crossover operation to generate new layer skipping combinations; mutation operation: randomly change certain genes of individuals with a certain probability; Determine the optimal solution: Select the layer skipping combination with the shortest average inference time from the final population as the optimal layer skipping combination.

3. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method of performing model pruning losslessly to accelerate large language model reasoning as described in claim 1.

4. An electronic device, characterized in that: It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method of performing model pruning losslessly to accelerate large language model reasoning as described in claim 1.

Images