Bidirectional block floating point-based large language model reasoning acceleration method

Abstract

The invention is applicable to the technical field of computers, and provides a big language model reasoning acceleration method based on bidirectional block floating points, which comprises the following steps: encoding an input text into a Token sequence, representing hidden representation and logits in a bidirectional block floating point number format in a Transform reasoning process, and combining a Softmax normalization method of a table look-up method based on bidirectional block floating points to obtain a big language model reasoning acceleration model; and low-bit efficient reasoning is realized, and a reasoning result is generated. According to the method, the calculation complexity and the storage overhead of the reasoning process are remarkably reduced while the generation precision is kept, and the reasoning speed and the energy efficiency ratio of the large language model are improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a method, apparatus, terminal device and storage medium for accelerating large language model inference based on bidirectional block floating point. Background Technology

[0002] In the field of natural language processing, significant progress has been made in text generation tasks with the widespread application of large language models (LLMs) such as GPT and BERT. These models rely on large-scale parameter training and possess powerful contextual understanding and generation capabilities, and are widely used in scenarios such as dialogue systems, automatic writing, and code generation. However, due to the large model size, deep inference paths, and high parameter dimensionality, computational resource consumption and response latency have become major bottlenecks in practical deployments, necessitating effective inference acceleration methods.

[0003] Existing methods for accelerating large model inference primarily reduce computational overhead through model quantization, pruning, and distillation. Quantization methods often employ fixed-point integers or low-order floating-point formats (such as INT8 and FP8) to compress model parameters and intermediate representations. However, these methods rely on high-precision floating-point calculations or complex hardware support in nonlinear computation stages (such as Softmax, activation functions, and LayerNorm), which can easily lead to decreased accuracy or complex hardware implementation, making it difficult to achieve truly efficient end-to-end inference while maintaining model performance. Summary of the Invention

[0004] This application provides a method, apparatus, terminal device, and storage medium for accelerating large language model inference based on bidirectional block floating-point. It can solve the problem that current methods for accelerating large model inference rely on high-precision floating-point calculations or complex hardware support in the nonlinear calculation stage, which can easily lead to a decrease in accuracy or complex hardware implementation, and make it difficult to achieve truly efficient end-to-end inference while ensuring model performance.

[0005] In a first aspect, embodiments of this application provide a method for accelerating large language model inference based on bidirectional block floating-point numbers, comprising: S1, segmenting the input text to be processed into words to obtain corresponding token sequences; S2, mapping the token sequences to word embedding vectors and performing positional encoding with corresponding positional information to obtain initial hidden representations; S3, inputting the initial hidden representations into a multi-layer Transformer to obtain the final hidden representations for each input token, wherein the format of the final hidden representations is a bidirectional block floating-point number; S4, mapping the final hidden representations to a vocabulary dimension through a linear transformation to obtain... S5. Based on the lookup table method of bidirectional block floating-point numbers, perform Softmax normalization on the logits to obtain the probability distribution of each candidate token in the vocabulary; S6. According to the probability distribution and combined with a greedy strategy, select the next output token; S7. Concatenate the next output token to the end of the generated text sequence, and repeat steps S3 to S6 in an autoregressive manner until the preset termination condition is met; S8. Merge the sub-words in the generated token sequence, concatenate the sub-words into complete words and remove all special control characters to generate the inference result.

[0006] In one possible implementation of the first aspect, the multi-layer Transformer includes N Transformers, where N is a positive integer. S3 above inputs the initial hidden representation into the multi-layer Transformer to obtain the final hidden representation for each input Token, including:

[0007] The initial hidden representation is input into the first Transformer to generate the first hidden representation;

[0008] The first hidden representation is input into the second Transformer to generate the second hidden representation;

[0009] Repeat the above steps until the (N-1)th hidden representation is input into the Nth Transformer to generate the final hidden representation.

[0010] Optionally, in another possible implementation of the first aspect, the above-mentioned inputting the initial hidden representation into the first Transformer to generate the first hidden representation includes:

[0011] For the weight matrix W Q 、W K 、W V Convert to a two-way block floating-point format;

[0012] The initial hidden representation is compared with the weight matrix W. Q 、W K 、WV Perform linear multiplication and addition operations to generate Query vector, Key vector, and Value vector;

[0013] Based on the query vector and key vector, the attention score is calculated and converted into a bidirectional block floating-point number format;

[0014] For each attention score, a lookup table based on bidirectional block floating-point numbers is used to generate the corresponding attention weights;

[0015] The attention weights and the value vector are weighted and summed to output the attention representation;

[0016] The attention representation is processed by layer normalization using a lookup table method based on bidirectional block floating-point numbers, and the output is a normalized representation;

[0017] The normalized representation is input into the feedforward neural network, and the feedforward network representation is output.

[0018] The feedforward network representation is processed by layer normalization using a lookup table method based on bidirectional block floating-point numbers, and the first hidden representation is output.

[0019] Optionally, in another possible implementation of the first aspect, the format of the aforementioned bidirectional block floating-point number includes a 1-bit sign bit, a 1-bit flag bit, multiple bits of mantissa bits, multiple bits of shared exponent bits, and multiple bits of overlap bits; the sign bit is used to indicate the positive or negative value, the flag bit is used to mark whether the original mantissa is shifted left or right, the mantissa bits are divided into high and low mantissa arrays according to the left and right shifts, the high mantissa array is formed by shifting the original mantissa to the left, and the low mantissa array is formed by shifting the original mantissa to the right, the shared exponent bit is determined by the number with the largest absolute value among all values ​​in the current data block, and serves as the reference exponent for all values ​​in the block, and the overlap bits are included in the mantissa bits to indicate the small changes in the mantissa between adjacent value quantization boundaries, so as to reduce the truncation error caused by bit width compression.

[0020] Optionally, in another possible implementation of the first aspect, the above-described table lookup method based on bidirectional block floating-point numbers includes:

[0021] Convert block input data into bidirectional block floating-point numbers;

[0022] Select the corresponding lookup table segment based on the shared index of the bidirectional block floating-point number;

[0023] The approximate result of the nonlinear function is obtained by performing a table lookup in segments of the selected lookup table based on the mantissa and flag bits of the bidirectional block floating-point number.

[0024] The approximate result of the nonlinear function is input into the encoder for encoding, and the calculation result is generated.

[0025] Secondly, embodiments of this application provide a large language model inference acceleration device based on bidirectional block floating-point numbers, comprising: a first acquisition module, configured to segment the input text to be processed into words and acquire the corresponding token sequence; a second acquisition module, configured to map the token sequence into word embedding vectors and perform position encoding with corresponding position information to acquire an initial hidden representation; a third acquisition module, configured to input the initial hidden representation into a multi-layer Transformer to acquire the final hidden representation of each input token, wherein the format of the final hidden representation is a bidirectional block floating-point number; and a fourth acquisition module, configured to map the final hidden representation to a vocabulary dimension through a linear transformation. The system performs the following steps: First, it obtains the corresponding logits. Second, it uses a lookup table method based on bidirectional block floating-point numbers to perform Softmax normalization on the logits, obtaining the probability distribution of each candidate token in the vocabulary. Third, it uses a selection module to select the next output token based on the probability distribution and a greedy strategy. Fourth, it uses a concatenation module to concatenate the next output token to the end of the generated text sequence, repeating steps S3 to S6 in an autoregressive manner until a preset termination condition is met. Fifth, it uses a generation module to merge sub-words in the generated token sequence, concatenating the sub-words into complete words and removing all special control characters to generate the inference result.

[0026] Thirdly, embodiments of this application provide a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method for accelerating large language model inference based on bidirectional block floating-point.

[0027] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned method for accelerating large language model inference based on bidirectional block floating-point.

[0028] In this application, the input text is encoded into a token sequence, and the hidden representation and logits are represented in a bidirectional block floating-point format during Transformer inference. Combined with a Softmax normalization method based on a bidirectional block floating-point lookup table, low-bit, high-efficiency inference is achieved, generating inference results. This application significantly reduces the computational complexity and storage overhead of the inference process while maintaining generation accuracy, thus improving the inference speed and energy efficiency of large language models. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating a method for accelerating large language model inference based on bidirectional block floating-point provided in an embodiment of this application;

[0031] Figure 2 This is a schematic diagram of the structure of a bidirectional block floating-point number provided in an embodiment of this application;

[0032] Figure 3 This is a flowchart illustrating the process of converting a floating-point number to a bidirectional block floating-point number according to an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of the structure of a large language model inference acceleration device based on bidirectional block floating-point provided in an embodiment of this application;

[0034] Figure 5 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0035] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0036] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0037] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0038] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0039] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] The following description, with reference to the accompanying drawings, details a method, apparatus, terminal device, and storage medium for accelerating large language model inference based on bidirectional block floating-point.

[0042] Figure 1 The diagram shows a flowchart of a method for accelerating large language model inference based on bidirectional block floating-point provided in an embodiment of this application.

[0043] like Figure 1 As shown, this method for accelerating inference in large language models based on bidirectional block floating-point operations, applied to LLM models, includes the following steps:

[0044] S1. Segment the input text to be processed into words and obtain the corresponding token sequence;

[0045] S2. Map the token sequence to a word embedding vector and perform positional encoding with the corresponding positional information to obtain the initial hidden representation;

[0046] S3. Input the initial hidden representation into the multi-layer Transformer to obtain the final hidden representation of each input Token. The format of the final hidden representation is a bidirectional block floating-point number.

[0047] Furthermore, in this embodiment, the multi-layer Transformer includes N Transformers, where N is a positive integer. The above-described S3 inputs the initial hidden representation to the multi-layer Transformer to obtain the final hidden representation for each input Token, including:

[0048] S31. Input the initial hidden representation into the first Transformer to generate the first hidden representation;

[0049] S32. Input the first hidden representation into the second Transformer to generate the second hidden representation;

[0050] S33. Repeat steps S31-S32 until the (N-1)th hidden representation is input into the Nth Transformer to generate the final hidden representation.

[0051] Furthermore, in this embodiment, the above-mentioned S31, inputting the initial hidden representation into the first Transformer to generate the first hidden representation, includes:

[0052] For the weight matrix W Q 、W K 、W V Convert to a two-way block floating-point format;

[0053] The initial hidden representation is compared with the weight matrix W. Q 、W K 、W V Perform linear multiplication and addition operations to generate Query vector, Key vector, and Value vector;

[0054] Based on the query vector and key vector, the attention score is calculated and converted into a bidirectional block floating-point number format;

[0055] For each attention score, a lookup table based on bidirectional block floating-point numbers is used to generate the corresponding attention weights;

[0056] The attention weights and the value vector are weighted and summed to output the attention representation;

[0057] The attention representation is processed by layer normalization using a lookup table method based on bidirectional block floating-point numbers, and the output is a normalized representation;

[0058] The normalized representation is input into the feedforward neural network, and the feedforward network representation is output.

[0059] The feedforward network representation is processed by layer normalization using a lookup table method based on bidirectional block floating-point numbers, and the first hidden representation is output.

[0060] It should be noted that the specific methods of step S31 are the same as those of steps S32 and S33.

[0061] It should be noted that due to the limited range of integer representation, block floating-point numbers possess the range of floating-point numbers while largely preserving the computational efficiency of integers. Therefore, block floating-point numbers are considered to have the potential to replace integers. For a set of block floating-point numbers, their actual values ​​are as follows:

[0062]

[0063] Among them, e m represents the shared exponent (here, the maximum exponent within the block), m represents the mantissa, and s represents the sign bit.

[0064] The bidirectional block floating-point number proposed in this application is an optimization of block floating-point numbers, reducing the indexing error caused by indexing the maximum value within the block. The quantization error of the block floating-point number has a mean of 0, and the variance is shown below:

[0065]

[0066] Among them, L m N represents the length of the mantissa. y The bit width is exponentially related to the shared index. This represents the probability density function of the shared exponent. As can be seen from the above formula, the greater the probability of choosing the maximum value as the shared exponent, the greater the variance of the quantization error of the block floating-point number. To reduce the quantization error of block floating-point numbers, a 1-bit flag is added to the block floating-point number, giving it the ability to point to the middle exponent and reducing the probability of choosing the maximum value as the shared exponent.

[0067] Specifically, such as Figure 2 As shown, the bidirectional block floating-point number format proposed in this application includes a 1-bit sign bit, a 1-bit flag bit, multiple bits of mantissa bits, multiple bits of shared exponent bits, and multiple bits of overlap bits. The sign bit is used to indicate the positive or negative value, the flag bit is used to mark whether the original mantissa is shifted left or right, the mantissa bits are divided into high and low mantissa arrays according to the left and right shifts, the high mantissa array is formed by shifting the original mantissa to the left, and the low mantissa array is formed by shifting the original mantissa to the right. The shared exponent bit is determined by the number with the largest absolute value among all values ​​in the current data block, and serves as the reference exponent for all values ​​in the block. The overlap bits are included in the mantissa bits and are used to indicate the small changes in the mantissa between adjacent value quantization boundaries, so as to reduce the truncation error caused by bit width compression.

[0068] As one possible implementation method, Figure 3This demonstrates the conversion process from traditional floating-point numbers to bidirectional block floating-point numbers with a 4-bit mantissa and a 5-bit shared exponent. Thanks to the lossless capture of outliers and low quantization error of bidirectional block floating-point numbers, they can be directly applied to the quantization of linear layer activations and weights in large language models without fine-tuning or calibration. The dot product formula for bidirectional block floating-point numbers can then be written as follows:

[0069]

[0070] Furthermore, the table lookup method based on bidirectional block floating-point numbers proposed in this application has the following specific steps:

[0071] Convert block input data into bidirectional block floating-point numbers;

[0072] Select the corresponding lookup table segment based on the shared index of the bidirectional block floating-point number;

[0073] The approximate result of the nonlinear function is obtained by performing a table lookup in segments of the selected lookup table based on the mantissa and flag bits of the bidirectional block floating-point number.

[0074] The approximate result of the nonlinear function is input into the encoder for encoding, and the calculation result is generated.

[0075] S4. Map the final hidden representation to the vocabulary dimension through a linear transformation and obtain the corresponding logits;

[0076] S5. Based on the lookup method of bidirectional block floating-point numbers, perform Softmax normalization on logits to obtain the probability distribution of each candidate token in the vocabulary.

[0077] S6. Based on the probability distribution and combined with a greedy strategy, select the next output token;

[0078] S7. Patch the next output token to the end of the generated text sequence, and repeat steps S3 to S6 in an autoregressive manner until the preset termination condition is met.

[0079] S8. Merge the sub-words of the generated Token sequence, concatenate the sub-words into complete words and remove all special control characters to generate the reasoning result.

[0080] For example, the novel data format proposed in this application reduces the probability of selecting a maximum value as the shared exponent by adding a one-bit flag, thus achieving lower quantization error. For linear layers: with a mantissa width of 6 bits, near-lossless quantization is achieved for three mainstream open-source models: OPT, LlaMA, and Qwen. With a mantissa width of 4 bits, the average precision loss on the three mainstream open-source models is less than 7%.

[0081] For nonlinear layers: with a mantissa bit width of 10, the accuracy loss caused by SiLu, Softmax, and Layernorm quantization in large language models is about 3%.

[0082] This application provides a method for accelerating large language model inference based on bidirectional block floating-point numbers. By encoding the input text into a token sequence and representing the hidden representation and logits in bidirectional block floating-point format during Transformer inference, combined with a Softmax normalization method based on a lookup table of bidirectional block floating-point numbers, low-bit, high-efficiency inference is achieved, generating inference results. This application significantly reduces the computational complexity and storage overhead of the inference process while maintaining generation accuracy, improving the speed and energy efficiency of large language model inference.

[0083] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0084] Corresponding to the above embodiment is a method for accelerating large language model inference based on bidirectional block floating-point, Figure 4 The diagram shows a structural block diagram of a large language model inference acceleration device based on bidirectional block floating-point provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0085] Reference Figure 4 The device 400 includes:

[0086] The first acquisition module 401 is used to segment the input text to be processed into words and obtain the corresponding Token sequence.

[0087] The second acquisition module 402 is used to map the Token sequence into a word embedding vector and perform position encoding with the corresponding position information to obtain the initial hidden representation;

[0088] The third acquisition module 403 is used to input the initial hidden representation into the multi-layer Transformer and obtain the final hidden representation of each input Token. The format of the final hidden representation is a bidirectional block floating-point number.

[0089] The fourth acquisition module 404 is used to map the final hidden representation to the vocabulary dimension through a linear transformation and obtain the corresponding logits;

[0090] The fifth acquisition module 405 is used to perform Softmax normalization on logits according to the lookup table method based on bidirectional block floating-point numbers, and obtain the probability distribution of each candidate token in the vocabulary.

[0091] Select module 406 to select the next output token based on the probability distribution and in combination with a greedy strategy;

[0092] The splicing module 407 is used to splice the next output token to the end of the generated text sequence, and repeatedly execute the data transmission process between the third acquisition module 403 and the selection module 406 in an autoregressive manner until the preset termination condition is met.

[0093] The generation module 408 is used to merge sub-words in the generated token sequence, concatenate the sub-words into complete words and remove all special control characters to generate inference results.

[0094] In practical use, the large language model inference acceleration device based on bidirectional block floating-point provided in this application embodiment can be configured in any terminal device to execute the aforementioned large language model inference acceleration method based on bidirectional block floating-point.

[0095] This application provides a large language model inference acceleration device based on bidirectional block floating-point numbers. First, the input text is encoded as a token sequence, and then the hidden representation and logits are represented in bidirectional block floating-point format during Transformer inference. Combined with a Softmax normalization method based on a bidirectional block floating-point lookup table, low-bit, high-efficiency inference is achieved to generate the inference result. This application significantly reduces the computational complexity and storage overhead of the inference process while maintaining generation accuracy, thus improving the inference speed and energy efficiency of large language models.

[0096] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0098] To implement the above embodiments, this application also proposes a terminal device.

[0099] Figure 5 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application.

[0100] like Figure 5 As shown, the terminal device 200 includes:

[0101] The system includes a memory 210 and at least one processor 220, and a bus 230 connecting different components (including the memory 210 and the processor 220). The memory 210 stores a computer program, and when the processor 220 executes the program, it implements the method for accelerating large language model inference based on bidirectional block floating-point as described in the embodiments of this application.

[0102] Bus 230 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0103] Terminal device 200 typically includes various electronically readable media. These media can be any available media that can be accessed by terminal device 200, including volatile and non-volatile media, removable and non-removable media.

[0104] Memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. Terminal device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 260 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 230 via one or more data media interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0105] A program / utility 280 having a set (at least one) of program modules 270 may be stored in, for example, memory 210. Such program modules 270 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 270 typically perform the functions and / or methods described in the embodiments of this application.

[0106] Terminal device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and with one or more devices that enable a user to interact with terminal device 200, and / or with any device that enables terminal device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 292. Furthermore, terminal device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 293. As shown, network adapter 293 communicates with other modules of terminal device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with terminal device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0107] The processor 220 performs various functional applications and data processing by running programs stored in the memory 210.

[0108] It should be noted that the implementation process and technical principles of the terminal device in this embodiment are explained in the foregoing description of a method for accelerating large language model inference based on bidirectional block floating point in this application embodiment, and will not be repeated here.

[0109] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0110] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0112] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0113] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for accelerating reasoning in large language models based on bidirectional block floating-point operations, characterized in that, When applied to an LLM model, the method includes: S1. Segment the input text to be processed into words and obtain the corresponding token sequence; S2. Map the token sequence to a word embedding vector and perform position encoding with the corresponding position information to obtain the initial hidden representation; S3. Input the initial hidden representation into a multi-layer Transformer to obtain the final hidden representation of each input Token. The format of the final hidden representation is a bidirectional block floating-point number. S4. Map the final hidden representation to the vocabulary dimension through a linear transformation to obtain the corresponding logits; S5. Based on the lookup table method of bidirectional block floating-point numbers, the logits are normalized by Softmax to obtain the probability distribution of each candidate token in the vocabulary. S6. Based on the probability distribution and in conjunction with a greedy strategy, select the next output token; S7. Patch the next output token to the end of the generated text sequence, and repeat steps S3 to S6 in an autoregressive manner until the preset termination condition is met. S8. Merge the sub-words of the generated Token sequence, concatenate the sub-words into complete words and remove all special control characters to generate the reasoning result.

2. The method as described in claim 1, characterized in that, The multi-layer Transformer includes N Transformers, where N is a positive integer. S3 inputs the initial hidden representation into the multi-layer Transformer to obtain the final hidden representation for each input Token, including: The initial hidden representation is input into the first Transformer to generate the first hidden representation; The first hidden representation is input into the second Transformer to generate the second hidden representation; Repeat the above steps until the (N-1)th hidden representation is input into the Nth Transformer to generate the final hidden representation.

3. The method as described in claim 2, characterized in that, The step of inputting the initial hidden representation into the first Transformer to generate the first hidden representation includes: For the weight matrix W Q 、W K 、W V Convert to a two-way block floating-point format; The initial hidden representation is respectively compared with the weight matrix W. Q 、W K 、W V Perform linear multiplication and addition operations to generate Query vector, Key vector, and Value vector; Based on the Query vector and the Key vector, an attention score is calculated and the attention score is converted into a bidirectional block floating-point number format; For each attention score, a lookup table based on bidirectional block floating-point numbers is used to generate the corresponding attention weight; The attention weights and the Value vector are weighted and summed to output the attention representation; The attention representation is processed by layer normalization using a lookup table method based on bidirectional block floating-point numbers, and a normalized representation is output. The normalized representation is input into the feedforward neural network, and the feedforward network representation is output. The feedforward network representation is processed by layer normalization using a lookup table method based on bidirectional block floating-point numbers, and the first hidden representation is output.

4. The method as described in claim 3, characterized in that, The format of the bidirectional block floating-point number includes a 1-bit sign bit, a 1-bit flag bit, multiple bits of mantissa bits, multiple bits of shared exponent bits, and multiple bits of overlap bits. The sign bit is used to indicate whether the value is positive or negative. The flag bit is used to mark whether the original mantissa is shifted left or right. The mantissa bits are divided into high and low mantissa arrays according to the left and right shifts. The high mantissa array is formed when the original mantissa is shifted left, and the low mantissa array is formed when the original mantissa is shifted right. The shared exponent bit is determined by the number with the largest absolute value among all values ​​in the current data block and serves as the reference exponent for all values ​​in the block. The overlap bits are included in the mantissa bits and are used to indicate the small changes in the mantissa between adjacent value quantization boundaries to reduce truncation errors caused by bit width compression.

5. The method as described in claim 4, characterized in that, The table lookup method based on bidirectional block floating-point numbers includes: Convert block input data into bidirectional block floating-point numbers; The corresponding lookup table segment is selected based on the shared index of the bidirectional block floating-point number; The approximate result of the nonlinear function is obtained by performing a table lookup in segments of the selected lookup table based on the mantissa and flag bits of the bidirectional block floating-point number. The approximate result of the nonlinear function is input into the encoder for encoding to generate the calculation result.

6. A large language model inference acceleration device based on bidirectional block floating-point, characterized in that, The device includes: The first acquisition module is used to segment the input text to be processed into words and obtain the corresponding Token sequence. The second acquisition module is used to map the Token sequence into a word embedding vector and perform position encoding with the corresponding position information to obtain the initial hidden representation; The third acquisition module is used to input the initial hidden representation into a multi-layer Transformer to obtain the final hidden representation of each input Token. The format of the final hidden representation is a bidirectional block floating-point number. The fourth acquisition module is used to map the final hidden representation to the vocabulary dimension through a linear transformation and obtain the corresponding logits; The fifth acquisition module is used to perform Softmax normalization on the logits according to the lookup table method based on bidirectional block floating-point numbers, and obtain the probability distribution of each candidate token in the vocabulary. The selection module is used to select the next output token based on the probability distribution and in combination with a greedy strategy. The splicing module is used to splice the next output token to the end of the generated text sequence, and repeat steps S3 to S6 in an autoregressive manner until the preset termination condition is met. The generation module is used to merge sub-words into the generated token sequence, concatenate the sub-words into complete words, remove all special control characters, and generate inference results.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images