Methods, systems, terminals, and media for accelerating inference in large-scale language models
By using FlashAttention and Lazy Batch-Updates techniques to segment and quantize large-scale language models, the problem of excessively long inference time is solved, achieving fast inference and high efficiency, making it suitable for various complex scenarios.
Patent Information
- Application Number
- CN202311436719.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-31
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-10-31
AI Technical Summary
The excessively long inference time of large-scale language models leads to a decline in user experience. Existing techniques such as model pruning, model distillation, and algorithm optimization have limitations in practical applications, necessitating new inference acceleration methods and systems.
The FlashAttention technique is used to divide the vector into blocks, and the floating-point model parameters of large-scale language models are converted into fixed-point numbers or low-bit-width integer representations. The Lazy Batch-Updates method is combined to balance computation and memory access.
It improves the inference speed and adaptability of large-scale language models, enabling real-time responses in complex scenarios and enhancing inference efficiency, especially under conditions of limited hardware resources.
Smart Images

Figure CN117574970B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer software technology, and specifically to a method, system, terminal, and medium for accelerating inference in large-scale language models. Background Technology
[0002] Large-scale language models (such as the GPT series) perform exceptionally well in various natural language processing tasks and are widely used in question-answering systems, chatbots, text generation, and other fields. However, as the model size increases, the inference time and computational resource costs also increase, posing challenges to practical applications. For example, when users expect to quickly obtain responses or analysis results from the model in real-time scenarios, excessively long inference times can lead to a degraded user experience.
[0003] The main solutions currently available can be categorized into three types: model pruning, model distillation, and algorithm optimization.
[0004] Model pruning: Model pruning methods reduce model size and inference time by removing less important parameters or neurons. These methods primarily focus on identifying and removing parameters that have minimal impact on the model's output. Advantages: Effectively reduces model size and improves inference speed. Disadvantages: Over-pruning can lead to performance degradation; the degree of pruning needs careful consideration.
[0005] Model distillation: Model distillation is a method for training a smaller model by using the knowledge of a larger model to guide the learning of the smaller model. In this process, the larger model acts as the "teacher," and the smaller model acts as the "student." Advantages: The resulting smaller model has faster inference speed while preserving as much of the performance of the larger model as possible. Disadvantages: It requires additional training, and the effectiveness of distillation can be affected by many factors.
[0006] Algorithm optimization: Algorithm optimization primarily focuses on improving the model's computational process and increasing computational efficiency. For example, it utilizes faster matrix multiplication algorithms or parallel computing techniques to accelerate model inference. Advantages: Directly targets the model's computational process, improving inference speed without altering the model's structure. Disadvantages: For certain models or application scenarios, the optimization space may be limited.
[0007] Although existing technologies have accelerated reasoning in large-scale language models to some extent, the diversity and challenges in practical applications make it essential to research and develop new methods and systems for accelerating reasoning. Summary of the Invention
[0008] To address the shortcomings of existing technologies, this invention provides a method and system for accelerating inference in large-scale language models, which can improve the inference speed, adaptability, and efficiency of large-scale language models.
[0009] In a first aspect, embodiments of the present invention provide a method for accelerating inference in large-scale language models, comprising:
[0010] Obtain the user-inputted data to be processed;
[0011] Convert the data to be processed into a vector;
[0012] The vector is processed by FlashAttention technology to obtain the processing result, and the processing result is sent to SRAM;
[0013] Convert the floating-point model parameters of a large-scale language model into fixed-point numbers or low-bit-width integer representations.
[0014] Optionally, the specific method for using FlashAttention technology to divide the vector into blocks to obtain the processing result includes:
[0015] Divide the vector into blocks;
[0016] Calculate the maximum value of each vector block separately;
[0017] The first function is obtained by subtracting the exponential term from the maximum value element by element.
[0018] The second function is obtained by summing the vectors obtained after subtraction.
[0019] Dividing the first function by the second function yields the normalized exponential function of the subvector;
[0020] Perform iterative calculations, update the global normalization exponent function with the normalization exponent function of the sub-vectors, and obtain the complete vector calculation result.
[0021] Optionally, specific methods for converting the floating-point model parameters of a large-scale language model into fixed-point or low-bit-width integer representations include:
[0022] The Lazy Batch-Updates approach balances computation and memory access by dividing the update operations of large-scale language model weights into blocks, processing a group of weights at a time.
[0023] Secondly, an inference acceleration system for large-scale language models provided by embodiments of the present invention includes: a data acquisition module, a data conversion module, an attention mechanism optimization module, and a model quantization module, wherein the data acquisition module is used to acquire user-input data to be processed;
[0024] The data conversion module is used to convert the data to be processed into a vector.
[0025] The attention mechanism optimization module uses FlashAttention technology to divide the vector into blocks to obtain the processing result, and sends the processing result into SRAM;
[0026] The model quantization module is used to convert the floating-point model parameters of a large-scale language model into fixed-point numbers or low-bit-width integer representations.
[0027] Optionally, the attention mechanism optimization module includes a block division unit and a computation unit, wherein the block division unit is used to divide the vector into blocks;
[0028] The computing unit is used to calculate the maximum value of each vector block, subtract the exponent term from the maximum value element by element to obtain the first function, sum the vectors obtained after the subtraction to obtain the second function, divide the first function by the second function to obtain the normalized exponent function of the sub-vector, perform iterative calculation, update the global normalized exponent function with the normalized exponent function of the sub-vector, and obtain the complete vector calculation result.
[0029] Optionally, the model quantization module includes a balancing unit and a grouping processing unit, wherein the balancing unit uses the LazyBatch-Updates method to balance computation and memory access;
[0030] The grouping processing unit is used to divide the update operation of the large-scale language model weights into blocks, processing one group of weights at a time.
[0031] Thirdly, an embodiment of the present invention provides a smart terminal, including a processor, an input device, an output device, and a memory. The processor is connected to the input device, the output device, and the memory, respectively. The memory is used to store a computer program, the computer program including program instructions, and the processor is configured to call the program instructions to execute the method described in the above embodiments.
[0032] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the methods described in the above embodiments.
[0033] The beneficial effects of this invention are:
[0034] This invention provides a method for accelerating inference in large-scale language models, improving the inference speed of such models. Even with massive model structures and parameter counts, it can quickly complete the inference process, providing real-time responses to users. It enhances the adaptability of large-scale language model inference, enabling it to handle inference requests in various complex scenarios, including inference under specific hardware environments or resource constraints. Furthermore, it improves the efficiency of large-scale language model inference, not only in single inference iterations but also significantly enhancing the overall efficiency of model inference in high-concurrency, large-data-volume scenarios.
[0035] This invention provides a reasoning acceleration system, terminal, and medium for large-scale language models, which are based on the same inventive concept and have the same beneficial effects as the aforementioned reasoning acceleration method for large-scale language models. Attached Figure Description
[0036] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.
[0037] Figure 1 A flowchart of a method for accelerating inference in large-scale language models provided in the first embodiment of the present invention is shown;
[0038] Figure 2 The flowchart of the method for segmenting the vector using FlashAttention technology according to the first embodiment of the present invention is shown.
[0039] Figure 3 A structural block diagram of an inference acceleration system for large-scale language models provided by another embodiment of the present invention is shown;
[0040] Figure 4 A structural block diagram of a smart terminal provided by another embodiment of the present invention is shown. Detailed Implementation
[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0042] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0043] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0044] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0045] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."
[0046] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.
[0047] like Figure 1 The diagram illustrates a flowchart of a method for accelerating inference in large-scale language models according to a first embodiment of the present invention. This method is applicable to inference acceleration systems for large-scale language models and includes the following steps:
[0048] S1: Obtain the user-inputted data to be processed;
[0049] S2: Convert the data to be processed into a vector;
[0050] S3: The vector is processed by using the flash attention technique to obtain the processing result, and the processing result is sent to SRAM;
[0051] S4: Converts the floating-point model parameters of a large-scale language model into fixed-point numbers or low-bit-width integer representations.
[0052] In practical applications, the inference acceleration method for large-scale language models executes steps S1-S4 in sequence, with the result of each step serving as input for the next, thus accelerating the entire inference process and achieving efficient inference acceleration for large-scale language models. Step S3 primarily optimizes the attention mechanism to improve its computational efficiency on GPU devices, enabling efficient use of hardware. Step S4 involves quantizing the parameters of the large-scale language model. The model parameters are in floating-point format, and quantization converts these floating-point parameters into fixed-point numbers or low-bit representations, such as 8-bit or 16-bit integers. This significantly reduces the model's storage and computational requirements, thereby accelerating the inference process while saving storage space and computational resources.
[0053] Specifically, taking the A100 (40G HBM) GPU as an example, the SRAM memory is distributed across 108 streaming multiprocessors, with a size of 20MB and a read / write speed of 19TB / s. The HBM cell size is 40GB, with a read / write speed of 1.5TB / s. It can be seen that SRAM is much faster than HBM, but its size is much smaller. During GPU computation, input is loaded from HBM into registers and SRAM, and then written back to HBM after computation. SRAM (Static Random-Access Memory) and HBM (High Bandwidth Memory) are both memory technologies used in computers and other digital devices, but they have different uses and characteristics. SRAM is faster and has a simpler circuit structure, typically using six transistors to store one bit of data. HBM is a stacked DRAM technology with higher bandwidth but slower speed.
[0054] In the Transformer model, a large number of matrix operations are performed in the self-attention module. The calculation method of self-attention is as follows: (1)
[0055]
[0056] Q, K, and V are three matrices of shape N*d, where N represents the length of the sequence and d represents the dimension.
[0057] Decompose formula (1) into formulas (2), (3), and (4):
[0058] S = QK T ∈R (N×N) (2)
[0059] P = softmax(S) ∈ R (N×N) (3)
[0060] O=PV∈R (N×d) (4)
[0061] The space complexity of both S and P is O(N). 2 In the above calculation process, the data transfer process between HBM and SRAM can be summarized as follows:
[0062] a) Load Q and K from HBM, calculate formula (2), and then write S into HBM.
[0063] b) Read S from HBM, calculate formula (3), and then write P into HBM.
[0064] c) Load P and V from HBM, calculate formula (4), and then write O into HBM.
[0065] As can be seen from the above process, calculating formula (1) requires multiple read and write operations on the HBM cell, and because the read and write speed of the HBM cell is relatively slow, the calculation time of formula (1) is long. To reduce this problem, we use the FlashAttention technique to divide the matrix involved in the calculation into blocks and send them into SRAM, thereby effectively utilizing the high-speed calculation characteristics of SRAM and reducing the number of read and write operations on the HBM.
[0066] like Figure 2 The diagram shows a flowchart of the method for segmenting the vector using FlashAttention technology in step S3. The specific method for obtaining the processing result by segmenting the vector using FlashAttention technology includes:
[0067] Divide the vector into blocks;
[0068] Calculate the maximum value of each vector block separately;
[0069] The first function is obtained by subtracting the exponential term from the maximum value element by element.
[0070] The second function is obtained by summing the vectors obtained after subtraction.
[0071] Dividing the first function by the second function yields the normalized exponential function of the subvector;
[0072] Perform iterative calculations, update the global normalization exponent function with the normalization exponent function of the sub-vectors, and obtain the complete vector calculation result.
[0073] Specifically, the difficulty in calculating self-attention in blocks lies in calculating softmax in blocks, because softmax requires the summation of all elements, such as formula (5) which is the method for calculating the softmax of a vector.
[0074]
[0075] Formula (5) contains an exponential term. When x i When the value is large, the value of the exponent is also large, which can easily lead to numerical overflow. Therefore, the following method is usually used for calculation.
[0076] m(x) = max([x1,x2,...,x...) B (6)
[0077] Where xi is the i-th component of vector x, and m(x) is the maximum value in vector x.
[0078]
[0079] f(x) is a new vector, each term of which is based on formula (6) with the maximum value m(x) subtracted from the exponent term. This prevents the occurrence of beneficial phenomena during the calculation process.
[0080] l(x)=∑ i f(x) i (8)
[0081] l(x) is the summation term in the denominator of the softmax formula. Therefore, the original softmax formula can be transformed into the following format:
[0082]
[0083] The following section introduces the core idea of block-based computation of softmax. Assume a vector of size 2B. Dividing it into two parts, we get x = [x (1) ,x (2) ],in At this point, we first calculate x. (1) Partially, x (1) Substituting into the above formula, we obtain the local softmax.
[0084]
[0085]
[0086] l(x (1) )=∑ i f(x (1) ) i (12)
[0087]
[0088] Since the goal of subtracting the maximum value in formula (7) is the maximum value of the entire vector x, while the goal of subtracting the maximum value in formula (11) is x (1) The maximum value corresponding to the component is called the local softmax. Similarly, the summation in formula (12) is also a local operation.
[0089] Thus, following the same process, for each component x of vector x... (i) By performing the above calculation iterations, the final result of the complete vector x can be obtained.
[0090] Quantization is the process of converting floating-point model parameters and calculations into a low-bit-width integer representation. This conversion can significantly reduce the model's storage requirements and computational complexity, making it more suitable for deployment on resource-constrained devices. Motivations for quantization include:
[0091] 1) Storage savings: By reducing the number of bits per parameter, the model size can be significantly reduced. 2) Computational speedup: Low-bit-width operations are generally faster than floating-point operations. 3) Energy savings: On many hardware devices, integer operations consume less energy than floating-point operations. Common quantization methods include: 1) Static quantization: Quantization is performed after model training is complete. This requires no data access, only the model's weights. 2) Dynamic quantization: Quantization is performed during the model's inference phase, typically involving the quantization of weights and activations. 3) Quantization-Aware Training (QAT): The effects of quantization are simulated during training, making the model more robust to quantization operations. Model quantization also faces several challenges: First, accuracy loss: Quantization may cause a slight decrease in model accuracy, especially when using very low bit widths. Second, implementation complexity: Quantization may require specific hardware and software support.
[0092] In this embodiment, GPTQ is used to quantize the model. GPTQ employs a direct quantization strategy, avoiding model retraining and thus demonstrating excellent time efficiency. This method can quickly reduce the parameters of large GPT models (e.g., GPT-3-175B) to a smaller bit width, thereby reducing storage requirements. Although GPTQ utilizes quantization, it successfully reduces the parameter bit width to 3 or 4 bits while sacrificing almost no model accuracy. This ensures that the compressed model maintains performance close to the original model, which is crucial for many application scenarios. GPTQ also demonstrates excellent performance in deep quantization scenarios, such as reducing weights to 2 bits or ternary representation. This shows that it can not only handle lower bit widths but also maintain good accuracy under deeper bit width compression.
[0093] Traditional quantization methods involve significant computational overhead with minimal memory access during each weight update. This prevents full utilization of the massive computing power of modern GPUs. This embodiment employs a LazyBatch-Updates approach to balance computation and memory access. The update operation for a large-scale language model weight is divided into blocks, processing one set of weights at a time. This method more effectively balances computation and memory access, reducing memory access frequency. Simultaneously, batch operations improve GPU utilization.
[0094] Table 1 shows the disk space occupied by the model after quantization with different precisions, and the inference speed on CPU and GPU respectively, in ms / token.
[0095] Table 1
[0096]
[0097] This invention provides a method for accelerating large-scale language model inference, which improves the inference speed of large-scale language models. Even with massive model structures and parameter volumes, it can quickly complete the inference process and provide real-time responses to users. It enhances the adaptability of large-scale language model inference, enabling it to handle inference requests in various complex scenarios, including inference under specific hardware environments or resource constraints. Furthermore, it improves the efficiency of large-scale language model inference, not only in single inference iterations but also significantly enhancing the overall efficiency of model inference in high-concurrency, large-data-volume scenarios.
[0098] In the first embodiment described above, a method for accelerating inference in large-scale language models is provided. Correspondingly, this application also provides a system for accelerating inference in large-scale language models. Please refer to... Figure 3 This is a structural block diagram of a reasoning acceleration system for large-scale language models provided in the second embodiment of the present invention. Since the device embodiment is basically similar to the method embodiment, it is described simply; relevant details can be found in the description of the method embodiment. The device embodiment described below is merely illustrative.
[0099] like Figure 3The diagram illustrates a structural block diagram of an inference acceleration system for large-scale language models. The system includes: a data acquisition module, a data transformation module, an attention mechanism optimization module, and a model quantization module. The data acquisition module acquires user-input data to be processed; the data transformation module converts the data into vectors; the attention mechanism optimization module uses FlashAttention technology to perform block processing on the vectors to obtain the processing results, which are then stored in SRAM; the model quantization module converts the floating-point model parameters of the large-scale language model into fixed-point numbers or low-bit-width integer representations. The system comprises four modules: data acquisition, data transformation, attention mechanism optimization, and model quantization. These four modules work together to achieve efficient inference acceleration for large-scale language models. The functions of each module and the interaction methods between them are briefly introduced below.
[0100] 1. Data Acquisition Module: Used to acquire user-input data to be processed.
[0101] 2. Data Conversion Module: Used to convert the data to be processed into vectors.
[0102] 3. Attention Mechanism Optimization Module:
[0103] The main task of this module is to improve the computational efficiency of the attention mechanism on GPU devices by optimizing its computational methods, thereby enabling efficient use of the hardware.
[0104] 4. Model Quantization Module:
[0105] The model quantization module performs quantization processing on the model's parameters. Typically, model parameters are in floating-point format, and quantization converts these floating-point parameters into fixed-point numbers or low-bit representations, such as 8-bit or 16-bit integers. This significantly reduces the model's storage and computational requirements, thereby accelerating the inference process and saving storage space and computing resources.
[0106] These four modules work together to form the system architecture of this invention embodiment. In practice, the process proceeds sequentially as data acquisition, data conversion, model format conversion, and model quantization. The result of each step serves as the input for the next step, thereby accelerating the entire inference process.
[0107] The attention mechanism optimization module includes a block division unit and a computation unit. The block division unit is used to divide the vector into blocks. The computation unit is used to calculate the maximum value of each block of vectors, subtract the exponent term from the maximum value element by element to obtain the first function, sum the vectors obtained after subtraction to obtain the second function, divide the first function by the second function to obtain the normalized exponent function of the sub-vector, perform iterative calculation, update the global normalized exponent function with the normalized exponent function of the sub-vector, and obtain the complete vector calculation result.
[0108] The model quantization module includes a balancing unit and a group processing unit. The balancing unit uses the Lazy Batch-Updates method to balance computation and memory access. The group processing unit is used to divide the update operation of the large-scale language model weights into blocks, processing a group of weights at a time.
[0109] This invention provides a large-scale language model inference acceleration system, which improves the inference speed of large-scale language models. Even with massive model structures and parameter volumes, it can quickly complete the inference process and provide users with real-time responses. It enhances the adaptability of large-scale language model inference, enabling it to handle inference requests in various complex scenarios, including inference under specific hardware environments or resource constraints. Furthermore, it improves the efficiency of large-scale language model inference, not only in single inference iterations but also significantly enhancing the overall efficiency of model inference in high-concurrency, large-data-volume scenarios.
[0110] like Figure 4 The diagram shows a structural block diagram of a smart terminal according to another embodiment of the present invention. The terminal includes a processor, an input device, an output device, and a memory. The processor is connected to the input device, the output device, and the memory. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions to execute the method described in the above embodiments.
[0111] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0112] Input devices may include touchpads, fingerprint sensors (for collecting the user's fingerprint information and fingerprint orientation information), microphones, etc., while output devices may include displays (LCDs, etc.), speakers, etc.
[0113] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.
[0114] In specific implementations, the processor, input device, and output device described in the embodiments of the present invention can execute the implementation methods described in the method embodiments of the present invention, or they can execute the implementation methods described in the system embodiments of the present invention, which will not be repeated here.
[0115] The present invention also provides an embodiment of a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the methods described in the above embodiments.
[0116] The computer-readable storage medium can be an internal storage unit of the terminal described in the foregoing embodiments, such as the terminal's hard drive or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the terminal. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.
[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0118] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0119] In the several embodiments provided in this application, it should be understood that the disclosed terminals and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.
[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.
Claims
1. A method for inference acceleration of large-scale language models, characterized in that, include: Obtain the user-inputted data to be processed; Convert the data to be processed into a vector; The vector is processed by FlashAttention technology to obtain the processing result, and the processing result is sent to SRAM; Convert the floating-point model parameters of large-scale language models into fixed-point numbers or low-bit-width integer representations; The specific method for obtaining the processing result by using FlashAttention technology to divide the vector into blocks includes: Divide the vector into blocks; Calculate the maximum value of each vector block separately; The first function is obtained by subtracting the exponential term from the maximum value element by element. The second function is obtained by summing the vectors obtained after subtraction. Dividing the first function by the second function yields the normalized exponential function of the subvector; Perform iterative calculations, update the global normalization exponent function with the normalization exponent function of the sub-vectors, and obtain the complete vector calculation result.
2. The method of claim 1, wherein, The specific methods for converting the floating-point model parameters of a large-scale language model into fixed-point numbers or low-bit-width integer representations include: The Lazy Batch-Updates approach balances computation and memory access by dividing the update operations of large-scale language model weights into blocks, processing a group of weights at a time.
3. A reasoning acceleration system for large-scale language models, characterized in that, include: The system includes a data acquisition module, a data transformation module, an attention mechanism optimization module, and a model quantization module. The data acquisition module is used to acquire user-input data to be processed. The data conversion module is used to convert the data to be processed into a vector. The attention mechanism optimization module uses FlashAttention technology to divide the vector into blocks to obtain the processing result, and sends the processing result into SRAM; The model quantization module is used to convert the floating-point model parameters of a large-scale language model into fixed-point numbers or low-bit-width integer representations. The attention mechanism optimization module includes a block division unit and a computation unit, wherein the block division unit is used to divide the vector into blocks; The computing unit is used to calculate the maximum value of each vector block, subtract the exponent term from the maximum value element by element to obtain the first function, sum the vectors obtained after the subtraction to obtain the second function, divide the first function by the second function to obtain the normalized exponent function of the sub-vector, perform iterative calculation, update the global normalized exponent function with the normalized exponent function of the sub-vector, and obtain the complete vector calculation result.
4. The system of claim 3, wherein, The model quantization module includes a balancing unit and a group processing unit. The balancing unit uses the Lazy Batch-Updates method to balance computation and memory access. The grouping processing unit is used to divide the update operation of the large-scale language model weights into blocks, processing one group of weights at a time.
5. A smart terminal comprising a processor, an input device, an output device and a memory, the processor being connected with the input device, the output device and the memory respectively, the memory being used for storing a computer program, the computer program comprising program instructions, characterized in that, The processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-2.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-2.
Citation Information
Patent Citations
Data retrieval method and system based on large language model
CN116701431A
