Large language model reasoning acceleration method, device and system and medium
By performing offline quantization and dynamic adjustment of large language models, the problem of slow inference speed of large language models is solved, high-precision and low-cost inference acceleration is achieved, hardware resource requirements are reduced and hallucination problems are avoided.
Patent Information
- Application Number
- CN202411210630.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-03
AI Technical Summary
The inference speed of existing large language models is slow, resulting in increased hardware costs and it is difficult to achieve high-precision and low-cost model inference acceleration.
By performing offline quantization of the large language model, a quantization model is obtained, and multiple candidate word elements are inferred side by side in each decoding step for verification, and finally a complete output sequence is generated, and the quantization accuracy and hyperparameters are dynamically adjusted to adapt to hardware resources. and precision requirements.
The inference acceleration with high precision and low hardware cost is achieved, which reduces the hallucination problems arising from different models, improves the inference speed, and reduces the hardware resource requirements.
Smart Images

Figure CN119067221A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of financial technology, and in particular to a large language model reasoning acceleration method, device, system and medium. Background Art
[0002] At present, the development and iteration of large language model-related technologies are extremely rapid, and they are most commonly used in the field of intelligent dialogue. For example, companies can integrate industry resources to build a one-stop service platform covering "car insurance, car services, and car life" to provide comprehensive services to car owners. Car owners can obtain services including vehicle maintenance, violation inquiries, road rescue, parking payment, vehicle insurance, electronic insurance policy viewing, and annual inspection agency through the platform. However, the cost behind providing large and comprehensive services is to increase the speed at which users reach the target service. For most users, they often only need to use a small part of the functions, so intelligent dialogue becomes the user's first choice. The platform needs to feedback accurate answers based on the user's questions, including basic function entrances and answers to knowledge questions.
[0003] Since users are very sensitive to the speed at which large language models generate answers, and the effect of large language models is positively correlated with the number of their parameters, in order to improve the reasoning speed of large language models, the number of model parameters often increases sharply, increasing hardware costs and making it difficult to achieve high-precision and low-cost model reasoning acceleration. Summary of the invention
[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a large language model reasoning acceleration method, device, system and medium that can be applied to financial technology or other related fields. Its main purpose is to reduce hardware resource costs without reducing model performance, and to achieve high-precision and low-cost reasoning acceleration.
[0005] The technical solution of the present invention is as follows:
[0006] A first aspect of the present invention provides a large language model reasoning acceleration method, comprising:
[0007] Perform offline quantization processing on the large language model to obtain a quantized model;
[0008] Inferring multiple candidate word units in parallel in each decoding step according to the input sequence using the quantization model;
[0009] Inputting the plurality of candidate word-grams into the large language model for verification;
[0010] The verified candidate word-grams are taken as the output of the corresponding decoding step until the complete output sequence is generated.
[0011] In one embodiment, before performing offline quantization processing on the large language model to obtain the quantized model, the method further includes:
[0012] In response to the precision adjustment instruction, the quantization precision of the quantization model is dynamically adjusted.
[0013] In one embodiment, dynamically adjusting the quantization precision of the quantization model in response to the precision adjustment instruction includes:
[0014] In response to the precision adjustment instruction, confirming hardware resource information of the current reasoning environment;
[0015] According to the hardware resource information, the quantization accuracy of the quantization model is adaptively adjusted to the target accuracy according to a preset adjustment strategy.
[0016] In one embodiment, dynamically adjusting the quantization precision of the quantization model in response to the precision adjustment instruction includes:
[0017] In response to the precision adjustment instruction, confirming the target precision in the precision adjustment instruction;
[0018] The quantization accuracy of the quantization model is adjusted to the target accuracy.
[0019] In one embodiment, the target precision is INT8, INT4 or INT3.
[0020] In one embodiment, before inputting the plurality of candidate word-grams into the large language model for verification, the method further includes:
[0021] In response to the hyperparameter adjustment instruction, the hyperparameters of the large language model are dynamically adjusted.
[0022] In one embodiment, the step of using the verified candidate word-grams as the output of the corresponding decoding step until a complete output sequence is generated includes:
[0023] Confirm the verification result of each candidate word in order;
[0024] If the verification passes, the candidate word is used as the output of the corresponding decoding step;
[0025] If the verification fails, the verification is terminated, and new word units are output from the corresponding decoding step through the large language model until a complete output sequence is generated.
[0026] A second aspect of the present invention provides a large language model reasoning acceleration device, comprising:
[0027] The quantization module is used to perform offline quantization processing on the large language model to obtain a quantized model;
[0028] An inference module, configured to infer multiple candidate word units in parallel in each decoding step according to the input sequence using the quantization model;
[0029] A verification module, used for inputting the plurality of candidate word-grams into the large language model for verification;
[0030] The decoding output module is used to output the verified candidate word as the output of the corresponding decoding step until a complete output sequence is generated.
[0031] A third aspect of the present invention provides a large language model reasoning acceleration system, the system comprising at least one processor; and
[0032] a memory communicatively connected to the at least one processor; wherein,
[0033] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned large language model reasoning acceleration method.
[0034] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned large language model reasoning acceleration method.
[0035] Beneficial effects: The present invention discloses a large language model reasoning acceleration method, device, system and medium. Compared with the prior art, the embodiment of the present invention performs offline quantization processing on the large language model to obtain a quantized model; uses the quantized model to infer multiple candidate word units in parallel in each decoding step according to the input sequence; inputs the multiple candidate word units into the large language model for verification; and uses the verified candidate word units as the output of the corresponding decoding step until a complete output sequence is generated. By performing offline quantization processing on the large language model, only a small amount of hardware resources are required to obtain the parallel candidate word unit inference of the input sequence by the quantized model, increase the parallelism of each decoding step to achieve reasoning acceleration, and while speeding up the reasoning speed, it can also reduce the illusion problem caused by different models, ensure the model accuracy, and achieve high-precision and low-hardware cost reasoning acceleration. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0037] Figure 1 A flow chart of a large language model reasoning acceleration method provided by an embodiment of the present invention;
[0038] Figure 2A flow chart of step S104 in the large language model reasoning acceleration method provided by an embodiment of the present invention;
[0039] Figure 3 A flow chart of precision adjustment in a large language model reasoning acceleration method provided by an embodiment of the present invention;
[0040] Figure 4 Another flowchart of precision adjustment in the large language model reasoning acceleration method provided by an embodiment of the present invention;
[0041] Figure 5 A schematic diagram of the functional modules of a large language model reasoning acceleration device provided by an embodiment of the present invention;
[0042] Figure 6 A schematic diagram of the hardware structure of a large language model inference acceleration system provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.
[0044] At present, the development and iteration of large language model-related technologies are extremely rapid, and they are most commonly used in the field of intelligent dialogue. For example, companies can integrate industry resources to build a one-stop service platform covering "car insurance, car services, and car life" to provide comprehensive services to car owners. Car owners can obtain services including vehicle maintenance, violation inquiries, road rescue, parking payment, vehicle insurance, electronic insurance policy viewing, and annual inspection agency through the platform. However, the cost behind providing large and comprehensive services is to increase the speed at which users reach the target service. For most users, they often only need to use a small part of the functions, so intelligent dialogue becomes the user's first choice. The platform needs to feedback accurate answers based on the user's questions, including basic function entrances and answers to knowledge questions.
[0045] Since users are very sensitive to the speed at which large language models generate answers, and the effect of large language models is positively correlated with the number of their parameters, in order to improve the reasoning speed of large language models, the number of model parameters often increases sharply, increasing hardware costs and making it difficult to achieve high-precision and low-cost model reasoning acceleration.
[0046] In order to solve the above problems, the present invention proposes a large language model reasoning acceleration method, please refer to Figure 1 , Figure 1A flowchart of an embodiment of the large language model reasoning acceleration method provided by the present invention. The system architecture of the large language model reasoning acceleration method provided in this embodiment may include a first terminal device, a second terminal device, a third terminal device, a network and a server. The network is used to provide a medium for a communication link between the first terminal device, the second terminal device, the third terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.
[0047] The user may use the first terminal device, the second terminal device, and the third terminal device to interact with the server through the network to receive or send messages, etc. Various communication client applications may be installed on the first terminal device, the second terminal device, and the third terminal device, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).
[0048] The first terminal device, the second terminal device, and the third terminal device may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0049] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device, the second terminal device, and the third terminal device (for example only). The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.
[0050] It should be noted that the large language model inference acceleration provided by the embodiment of the present invention can generally be performed by the first terminal device, the second terminal device or the third terminal device. Correspondingly, the large language model inference acceleration provided by the embodiment of the present invention can also be set in the first terminal device, the second terminal device or the third terminal device.
[0051] Alternatively, the large language model reasoning acceleration method provided by the embodiment of the present invention can generally be executed by a server. Accordingly, the large language model reasoning acceleration device provided by the embodiment of the present invention can generally be arranged in a server. The large language model reasoning acceleration method provided by the embodiment of the present invention can also be executed by a server or a server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server. Correspondingly, the large language model reasoning acceleration device provided by the embodiment of the present invention can also be arranged in a server or a server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server.
[0052] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0053] like Figure 1 As shown, the large language model reasoning acceleration method provided in this embodiment specifically includes the following steps:
[0054] S101, performing offline quantization processing on the large language model to obtain a quantized model.
[0055] In this embodiment, the reasoning output of the large language model is divided into two stages, the first stage is the offline quantization stage, and the second stage is the decoding output stage. In the first stage, the large language model is quantized by offline quantization technology to obtain a quantized model for auxiliary inference. Offline quantization technology is a method for quantizing the weights and activation values of a model after model training is completed. This technology does not require modification or retraining of the model during the training process, so it is relatively simple and easy to deploy. Quantization can reduce the computational complexity and memory requirements of the model, and can also accelerate the reasoning process of the model on specific hardware. In specific implementation, offline quantization processing with low hardware occupancy cost can be achieved through quantization methods or tools such as AWQ, GPTQ, Bitsandbytes, etc. AWQ (Adaptive Weight Quantization) focuses on quantizing the weights of the model while keeping the activation values in floating point format. This method can adaptively select quantization strategies according to the statistical characteristics of the weights to reduce the accuracy loss in the quantization process; GPTQ (Generalized Post-Training Quantization) is a general post-training quantization method that aims to convert a trained neural network model into a quantized model to improve the inference speed and reduce the model size. GPTQ usually involves determining the quantization range and quantization parameters, as well as fine-tuning the model to adapt to the quantization error; Bitsandbytes is an open source, Python-based library for quantization of deep learning models. It provides a variety of quantization strategies and tools and supports a variety of deep learning frameworks including PyTorch. The quantization tool Bitsandbytes can simplify the quantization process and support end-to-end quantization processing from training to deployment, thereby significantly improving the efficiency of quantization processing. Specifically, a corresponding quantization method can be flexibly selected according to the application scenario, target hardware, and specific requirements for accuracy and performance, etc., which is not limited in this embodiment.
[0056] Compared with the common auxiliary model that needs to be trained by knowledge distillation to assist the large language model, retraining by means of knowledge distillation not only consumes a considerable amount of hardware resources, but also the auxiliary model and the large language model are two independent models in the pre-training stage, and different effects will be produced when the model is updated by gradient descent. Therefore, the knowledge learned by the model is not the same, which is prone to hallucination problems. In this embodiment, the quantization process of the large language model can be completed in a very short time through offline quantization technology, and the resources required for offline quantization are almost negligible, which can significantly reduce the occupation of video memory, and because the quantization model and the large language model belong to the same model (trained by exactly the same gradient descent), the two models contain the same number of parameters and the same knowledge distribution, so the hallucination problem caused by different models can be effectively avoided. Therefore, the reasoning of the large language model is assisted by the quantization model to save hardware resources while improving the reasoning speed.
[0057] S102: Utilize the quantization model to infer multiple candidate word units in parallel in each decoding step according to the input sequence.
[0058] In this embodiment, in the second stage, i.e., the decoding output stage, the quantization model obtained by offline quantization infers multiple candidate word units in parallel in each decoding step according to the input sequence to increase the parallelism of the model calculation in each decoding step. A word unit refers to a single element in a text, which can be a word, a number, a symbol, or any other basic unit of text. In a large language model (LLM), the input and output of the model are usually composed of a series of tokens, each of which is a basic unit of model processing and generation. Since the autoregressive decoding commonly used in LLM can only generate one token (word unit) in each decoding step, the generation of each token requires repeated reading and writing of the huge parameters of LLM, which will lead to low utilization of GPU computing resources, and the generation time of the sequence increases linearly with the increase of the sequence length. After quantization, a smaller quantized model is obtained, which can be used to assist in the inference of the large language model. In each decoding step, the quantized model is used to infer the word units that may be generated in multiple future decoding steps of the large language model, namely candidate word units, in parallel at a low computing cost. If the inference is accurate enough, the quantized model can generate multiple ultimately adopted word units in parallel in a single decoding step, thereby increasing the parallelism of the model calculation in each decoding step, reducing the total number of decoding steps, thereby reducing the repeated reading and writing of LLM parameters and achieving inference acceleration.
[0059] S103: Input the multiple candidate word-grams into the large language model for verification.
[0060] In this embodiment, after a plurality of candidate word-grams are efficiently and in parallel generated by the quantization model, the candidate word-grams are input into the large language model for verification, i.e., the appropriateness and accuracy of the candidate word-grams inferred by the quantization model in the current context are evaluated. The specific verification evaluation can be performed through different strategies, for example, Greedy Decoding selects the Token with the highest probability as the verification result; Nucleus Sampling samples Tokens from the probability distribution, but only considers the first few Tokens whose cumulative probability exceeds a certain threshold. The corresponding verification evaluation strategy can be flexibly set according to the needs, and this embodiment does not limit this.
[0061] During the verification process, starting from the first candidate word, each word will go through a verification algorithm to determine whether it is accepted. When given multiple candidate words generated by the quantization model, LLM will assume that the prefix input is correct and generate a token for verification. For example, when verifying the third candidate token, LLM uses the first two verified tokens as prefix input. Taking greedy decoding as an example, when the prefix is used as input, LLM will generate a token with the highest probability. If the verification token generated by the LLM itself is the same as the third candidate token, it means that the third candidate token has passed the verification, that is, the candidate token is the result that LLM would have generated by itself. Therefore, through the verification process, it can be theoretically guaranteed that the decoding result inferred based on the quantization model is consistent with the large language model autoregressive decoding result, so as not to lose decoding quality and ensure model accuracy.
[0062] S104: Taking the candidate word that has passed the verification as the output of the corresponding decoding step until a complete output sequence is generated.
[0063] In this embodiment, based on the verification results of multiple candidate word units by the large language model, once a candidate word unit passes the verification, it is added to the context of the current decoding step to prepare for the next decoding step, that is, the verified candidate word unit is directly adopted by the large language model, and the process of speculation, verification, and context update is repeated until the termination condition is met, the end token is generated, or the preset decoding length is reached, then the large language model output finally generates a complete output sequence. By pre-generating multiple candidate word units through a quantized model obtained by quantization with very few hardware resources and handing them over to the large language model for verification, the parallelism of each decoding step is increased to achieve reasoning acceleration, thereby reducing the overall decoding time. While maintaining the quality of the generated text, the decoding speed is significantly improved, and the illusion problem caused by different models can be alleviated, ensuring the accuracy of the model, and achieving high-precision and low-hardware cost reasoning acceleration.
[0064] In one embodiment, Figure 2 As shown,
[0065] The candidate word that passes the verification is used as the output of the corresponding decoding step until a complete output sequence is generated, including:
[0066] S201, confirming the verification result of each candidate word in order;
[0067] S202, if the verification is successful, taking the candidate word as the output of the corresponding decoding step;
[0068] S203: If the verification fails, the verification is terminated, and new word units are output from the corresponding decoding step through the large language model until a complete output sequence is generated.
[0069] In this embodiment, the large language model verifies multiple candidate word-grams inferred and generated by the quantization model, and uses this to confirm the verification result of each candidate word-gram. If the verification is successful, the candidate word-gram is used as the output of the corresponding decoding step, that is, the successfully verified candidate word-gram is adopted. If a candidate word-gram fails to be verified, the subsequent verification is terminated, and a new word-gram is output from the corresponding decoding step through the large language model. Since the generation of the large language model usually depends on the context, the word-gram that fails to be verified may affect the generation of subsequent word-grams. To ensure the accuracy of the model, if a word-gram fails during the verification process, the large language model needs to fall back to the previous successful word-gram and regenerate the word-gram of the subsequent decoding step until a complete output sequence is generated, thereby ensuring the accuracy and fluency of the final generated text sequence.
[0070] In one embodiment, before performing offline quantization processing on the large language model to obtain the quantized model, the method further includes:
[0071] In response to the precision adjustment instruction, the quantization precision of the quantization model is dynamically adjusted.
[0072] In this embodiment, in order to adapt to the reasoning environment of different resources or different reasoning speed requirements, the quantization accuracy can be dynamically adjusted before the quantization process. That is, the quantization strategy is dynamically adjusted according to different requirements to ensure the reasoning speed and reduce the model size while maintaining the model accuracy as much as possible. For example, the commonly used large model parameter format is FP16, that is, a parameter is represented by a 16-bit, 2-byte floating point number. The more commonly used quantization formats are INT8 and INT4, among which INT8 is a common quantization bit width choice because it can maintain good model accuracy while reducing the model size and accelerating reasoning. INT8 is an 8-bit integer that can represent data ranging from -128 to 127, using 8 bits to store values, so its dynamic range is much smaller than the standard 32-bit floating point number (FP32), but still larger than INT4; due to the large bit width, INT8 can provide relatively high accuracy and is usually used for weight and activation value representation in quantization. INT4 is a 4-bit integer that can represent data ranging from -8 to 7. Due to its small bit width, INT4 has lower precision and may introduce more precision loss during quantization. Therefore, INT4 quantization is usually used in extreme model compression scenarios. Although the precision loss may be large, it can significantly reduce the storage and memory usage of the model and is suitable for devices with extremely limited resources. The quantization formats of INT8 and INT4 can reduce the size of the model to half / a quarter of the original model.
[0073] Through quantization, the hardware resources used are significantly reduced, and the time consumed by the offline quantization process is almost negligible. The quantization accuracy of the quantization model can be dynamically switched according to different usage scenarios. For scenarios with higher accuracy, a version with higher quantization accuracy can be used. For scenarios with higher inference speed requirements, more aggressive quantization solutions can be used, such as INT4, INT3, etc. By dynamically adjusting the quantization accuracy, the quantization model can achieve a more optimized balance between performance and accuracy under different application scenarios and data conditions.
[0074] In one embodiment, Figure 3 As shown, in response to the precision adjustment instruction, dynamically adjusting the quantization precision of the quantization model includes:
[0075] S301, in response to the precision adjustment instruction, confirming the hardware resource information of the current reasoning environment;
[0076] S302: According to the hardware resource information, the quantization accuracy of the quantization model is adaptively adjusted to the target accuracy according to a preset adjustment strategy.
[0077] In this embodiment, the quantization accuracy of the quantization model is automatically adjusted in an adaptive manner, that is, when receiving the accuracy adjustment instruction, the hardware resource information of the reasoning environment in which the current model is located is automatically obtained, such as the model and speed of the CPU, the number of cores, and the cache size; the model, memory size and type of the GPU, the number of CUDA cores or stream processors, and the supported computing power; the total capacity, type, and speed of the memory; the type, capacity, and interface type of the storage device; the network interface and bandwidth; the type and version of the dedicated hardware accelerator, the operating system, etc. By obtaining this hardware resource information, the reasoning performance of the model can be evaluated and optimized. For example, if the GPU has a large number of CUDA cores and a high memory capacity, it may be more suitable for performing large-scale parallel computing tasks. Similarly, high-speed RAM and SSD can reduce the time for model loading and data processing, so this hardware resource information is particularly important for selecting a suitable model and quantization strategy.
[0078] After obtaining the hardware resource information, the quantization accuracy of the quantization model is adaptively adjusted according to the preset adjustment strategy, and it is adaptively adjusted to the target accuracy, such as INT8, INT4 or INT3. In specific implementation, the performance indicators of CPU, GPU and other special hardware can be analyzed first to determine the computing power, memory bandwidth and storage speed; according to the context of model operation (such as cloud servers, mobile devices, embedded systems, etc.), the specific requirements for inference speed, model size and energy consumption can be determined; according to the computing power of the hardware and the needs of the application scenario, the appropriate quantization bit width (such as INT8, INT4, etc.) can be selected. For example, high-performance GPUs may be more supportive of INT8 quantization, while resource-constrained devices may require INT4 quantization; and the data distribution and sensitivity of different layers in the model can be further analyzed to determine which layers can use lower precision quantization and which layers need to maintain higher precision. For example, INT8 quantization can be used for weights in the model, and for activation values, INT4 quantization can be used if the hardware supports it. By automatically obtaining hardware resource information and adaptively adjusting the quantization accuracy of the quantization model, the quantization processing of the large language model can meet the needs of different hardware resources and application scenarios, thereby increasing the scalability of large language model inference acceleration and broadening the scope of application.
[0079] In one embodiment, Figure 4 As shown, in response to the precision adjustment instruction, dynamically adjusting the quantization precision of the quantization model includes:
[0080] S401, in response to a precision adjustment instruction, confirming a target precision in the precision adjustment instruction;
[0081] S402: Adjust the quantization accuracy of the quantization model to the target accuracy.
[0082] In this embodiment, the quantization accuracy of the quantization model is adjusted manually, that is, the user can input the accuracy adjustment instruction according to actual needs, and the accuracy adjustment instruction carries a determined target accuracy, such as INT8, INT4 or INT3, etc. When the system receives the accuracy adjustment instruction, it directly parses and confirms the target accuracy in the instruction, thereby instructing the system to adjust the quantization accuracy of the quantization model to the target accuracy during offline quantization processing. This embodiment, through clear accuracy requirements and a method of directly adjusting the quantization accuracy, can make rapid adjustments according to the needs of different application scenarios, improve the flexibility and availability of the model, and ensure that the model adjustment is consistent with the expected goal, avoid wasting resources on unnecessary accuracy optimization, effectively improve the efficiency of accuracy adjustment, and thus improve the efficiency of model reasoning acceleration.
[0083] In one embodiment, before inputting the plurality of candidate word-grams into the large language model for verification, the method further includes:
[0084] In response to the hyperparameter adjustment instruction, the hyperparameters of the large language model are dynamically adjusted.
[0085] In this embodiment, in the decoding output stage, the hyperparameters of the large language model can be further dynamically adjusted through hyperparameter adjustment instructions. The specific hyperparameters to be adjusted dynamically can include, for example, learning rate, beam search width, temperature parameters, etc., to balance the generation speed and accuracy. By adjusting the hyperparameters, the number of tokens generated by the quantization model at one time, i.e., the number of candidate word units, and the verification accuracy, etc., can be controlled, thereby controlling the speed and accuracy of the entire system generation, achieving more flexible reasoning acceleration, and meeting the performance requirements in different application scenarios.
[0086] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.
[0087] Another embodiment of the present invention provides a large language model reasoning acceleration device, such as Figure 5 As shown, the device 1 comprises:
[0088] A quantization module 11 is used to perform offline quantization processing on the large language model to obtain a quantized model;
[0089] An inference module 12, configured to infer multiple candidate word units in parallel in each decoding step according to the input sequence using the quantization model;
[0090] A verification module 13, used for inputting the plurality of candidate word-units into the large language model for verification;
[0091] The decoding output module 14 is used to use the verified candidate word-units as the output of the corresponding decoding step until a complete output sequence is generated.
[0092] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which are more suitable for describing the large language model reasoning acceleration execution process than the program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.
[0093] In one embodiment, the device 1 further comprises:
[0094] The precision adjustment module is used to dynamically adjust the quantization precision of the quantization model in response to the precision adjustment instruction.
[0095] In one embodiment, the precision adjustment module includes:
[0096] An environment confirmation unit, configured to confirm hardware resource information of a current reasoning environment in response to a precision adjustment instruction;
[0097] The adaptive adjustment unit is used to adaptively adjust the quantization accuracy of the quantization model to a target accuracy according to the hardware resource information and a preset adjustment strategy.
[0098] In one embodiment, the precision adjustment module includes:
[0099] an accuracy confirmation unit, configured to confirm a target accuracy in the accuracy adjustment instruction in response to the accuracy adjustment instruction;
[0100] The precision adjustment unit is used to adjust the quantization precision of the quantization model to the target precision.
[0101] In one embodiment, the target precision is INT8, INT4 or INT3.
[0102] In one embodiment, the device 1 further comprises:
[0103] A hyperparameter adjustment module is used to dynamically adjust the hyperparameters of the large language model in response to the hyperparameter adjustment instruction.
[0104] In one embodiment, the decoding output module 14 includes:
[0105] A result confirmation unit, used to confirm the verification result of each candidate word in order;
[0106] A first decoding output unit, configured to use the candidate word as the output of the corresponding decoding step if the verification passes;
[0107] The second decoding output unit is used to terminate the verification if the verification fails, and output new word units from the corresponding decoding step through the large language model until a complete output sequence is generated.
[0108] Another embodiment of the present invention provides a large language model reasoning acceleration system, such as Figure 6 As shown, the system 10 includes:
[0109] One or more processors 110 and memory 120, Figure 6 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.
[0110] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.
[0111] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the large language model reasoning acceleration method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, the large language model reasoning acceleration method in the above method embodiment is implemented.
[0112] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0113] One or more units are stored in the memory 120. When executed by one or more processors 110, the large language model reasoning acceleration method in any of the above method embodiments is executed, for example, the following steps are executed:
[0114] Perform offline quantization processing on the large language model to obtain a quantized model;
[0115] Inferring multiple candidate word units in parallel in each decoding step according to the input sequence using the quantization model;
[0116] Inputting the plurality of candidate word-grams into the large language model for verification;
[0117] The verified candidate word-grams are taken as the output of the corresponding decoding step until the complete output sequence is generated.
[0118] In one embodiment, before performing offline quantization processing on the large language model to obtain the quantized model, the method further includes:
[0119] In response to the precision adjustment instruction, the quantization precision of the quantization model is dynamically adjusted.
[0120] In one embodiment, dynamically adjusting the quantization precision of the quantization model in response to the precision adjustment instruction includes:
[0121] In response to the precision adjustment instruction, confirming hardware resource information of the current reasoning environment;
[0122] According to the hardware resource information, the quantization accuracy of the quantization model is adaptively adjusted to the target accuracy according to a preset adjustment strategy.
[0123] In one embodiment, dynamically adjusting the quantization precision of the quantization model in response to the precision adjustment instruction includes:
[0124] In response to the precision adjustment instruction, confirming the target precision in the precision adjustment instruction;
[0125] The quantization accuracy of the quantization model is adjusted to the target accuracy.
[0126] In one embodiment, the target precision is INT8, INT4 or INT3.
[0127] In one embodiment, before inputting the plurality of candidate word-grams into the large language model for verification, the method further includes:
[0128] In response to the hyperparameter adjustment instruction, the hyperparameters of the large language model are dynamically adjusted.
[0129] In one embodiment, the step of using the verified candidate word-grams as the output of the corresponding decoding step until a complete output sequence is generated includes:
[0130] Confirm the verification result of each candidate word in order;
[0131] If the verification passes, the candidate word is used as the output of the corresponding decoding step;
[0132] If the verification fails, the verification is terminated, and new word units are output from the corresponding decoding step through the large language model until a complete output sequence is generated.
[0133] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the large language model reasoning acceleration method in any of the above method embodiments is executed, for example, the following steps are performed:
[0134] Perform offline quantization processing on the large language model to obtain a quantized model;
[0135] Inferring multiple candidate word units in parallel in each decoding step according to the input sequence using the quantization model;
[0136] Inputting the plurality of candidate word-grams into the large language model for verification;
[0137] The verified candidate word-grams are taken as the output of the corresponding decoding step until the complete output sequence is generated.
[0138] In one embodiment, before performing offline quantization processing on the large language model to obtain the quantized model, the method further includes:
[0139] In response to the precision adjustment instruction, the quantization precision of the quantization model is dynamically adjusted.
[0140] In one embodiment, dynamically adjusting the quantization precision of the quantization model in response to the precision adjustment instruction includes:
[0141] In response to the precision adjustment instruction, confirming hardware resource information of the current reasoning environment;
[0142] According to the hardware resource information, the quantization accuracy of the quantization model is adaptively adjusted to the target accuracy according to a preset adjustment strategy.
[0143] In one embodiment, dynamically adjusting the quantization precision of the quantization model in response to the precision adjustment instruction includes:
[0144] In response to the precision adjustment instruction, confirming the target precision in the precision adjustment instruction;
[0145] The quantization accuracy of the quantization model is adjusted to the target accuracy.
[0146] In one embodiment, the target precision is INT8, INT4 or INT3.
[0147] In one embodiment, before inputting the plurality of candidate word-grams into the large language model for verification, the method further includes:
[0148] In response to the hyperparameter adjustment instruction, the hyperparameters of the large language model are dynamically adjusted.
[0149] In one embodiment, the step of using the verified candidate word-grams as the output of the corresponding decoding step until a complete output sequence is generated includes:
[0150] Confirm the verification result of each candidate word in order;
[0151] If the verification passes, the candidate word is used as the output of the corresponding decoding step;
[0152] If the verification fails, the verification is terminated, and new word units are output from the corresponding decoding step through the large language model until a complete output sequence is generated.
[0153] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.
[0154] In summary, the present invention discloses a large language model reasoning acceleration method, device, system and medium, the method includes: performing offline quantization processing on the large language model to obtain a quantization model; using the quantization model to infer multiple candidate word units in parallel in each decoding step according to the input sequence; inputting the multiple candidate word units into the large language model for verification; using the verified candidate word units as the output of the corresponding decoding step until a complete output sequence is generated. By performing offline quantization processing on the large language model, only a small amount of hardware resources are required to obtain the parallel candidate word unit inference of the input sequence by the quantization model, increasing the parallelism of each decoding step to achieve reasoning acceleration, while speeding up the reasoning speed, it can also reduce the illusion problem caused by different models, ensure the model accuracy, and achieve high-precision and low-hardware cost reasoning acceleration.
[0155] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.
[0156] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A large language model reasoning acceleration method, characterized in that: include: Perform offline quantization processing on the large language model to obtain a quantized model; Inferring multiple candidate word units in parallel in each decoding step according to the input sequence using the quantization model; Inputting the plurality of candidate word-grams into the large language model for verification; The verified candidate word-grams are taken as the output of the corresponding decoding step until the complete output sequence is generated.
2. The large language model reasoning acceleration method according to claim 1, characterized in that: The candidate word that passes the verification is used as the output of the corresponding decoding step until a complete output sequence is generated, including: Confirm the verification result of each candidate word in order; If the verification passes, the candidate word is used as the output of the corresponding decoding step; If the verification fails, the verification is terminated, and new word units are output from the corresponding decoding step through the large language model until a complete output sequence is generated.
3. The large language model reasoning acceleration method according to claim 1, characterized in that: Before performing offline quantization processing on the large language model to obtain the quantized model, the method further includes: In response to the precision adjustment instruction, the quantization precision of the quantization model is dynamically adjusted.
4. The large language model reasoning acceleration method according to claim 3, characterized in that: The dynamically adjusting the quantization precision of the quantization model in response to the precision adjustment instruction includes: In response to the precision adjustment instruction, confirming hardware resource information of the current reasoning environment; According to the hardware resource information, the quantization accuracy of the quantization model is adaptively adjusted to the target accuracy according to a preset adjustment strategy.
5. The large language model reasoning acceleration method according to claim 3, characterized in that: The dynamically adjusting the quantization precision of the quantization model in response to the precision adjustment instruction includes: In response to the precision adjustment instruction, confirming the target precision in the precision adjustment instruction; The quantization accuracy of the quantization model is adjusted to the target accuracy.
6. The large language model reasoning acceleration method according to claim 4 or 5, characterized in that: The target precision is INT8, INT4 or INT3.
7. The large language model reasoning acceleration method according to claim 1, characterized in that: Before inputting the plurality of candidate word-grams into the large language model for verification, the method further includes: In response to the hyperparameter adjustment instruction, the hyperparameters of the large language model are dynamically adjusted.
8. A large language model reasoning acceleration device, characterized in that: include: The quantization module is used to perform offline quantization processing on the large language model to obtain a quantized model; An inference module, configured to infer multiple candidate word units in parallel in each decoding step according to the input sequence using the quantization model; A verification module, used for inputting the plurality of candidate word-grams into the large language model for verification; The decoding output module is used to output the verified candidate word as the output of the corresponding decoding step until a complete output sequence is generated.
9. A large language model reasoning acceleration system, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large language model reasoning acceleration method described in any one of claims 1-7.
10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the large language model reasoning acceleration method described in any one of claims 1-7.
Citation Information
Cited By
Model effect evaluation system and method based on quantitative reasoning
CN119292892A
A model effect evaluation system and method based on quantitative reasoning
CN119292892B
Web large language model reasoning acceleration optimization method and device based on WebGPU
CN119338016A
Large model reasoning method and system based on adjacency matrix and speculative decoding
CN119721244A
Speculation decoding method and device based on large model, equipment and storage medium
CN119806649A
