Large language model inference fast sampling method and device, server and medium

Abstract

Embodiments of the present application disclose a large language model inference fast sampling method and device, a server and a medium. The method comprises: determining whether the incoming K value is less than the preset K value threshold; when the K value is less than the preset K value threshold, obtaining the candidate score of the Top-K candidate word according to the K value; dividing the candidate score of the Top-K candidate word by the Temperature to obtain the optimized candidate score of the Top-K candidate word; writing the optimized candidate score distribution of the Top-K candidate word into the GPU shared memory, and using the thread bundle level reduction, exchange and reduction of the optimized probability of the Top-K candidate word. The inference efficiency can be greatly improved without reducing the inference accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, and particularly relates to a large language model inference fast sampling method and device, a server and a medium. BACKGROUND

[0002] In the inference process of a large language model (LLM), sampling strategies (such as Top-K, Top-P, Min-P and Temperature) are widely used to control the diversity and quality of generated text. Current mainstream implementations are usually based on deep learning frameworks such as PyTorch, and the sampling logic is completed by calling standard operators one by one.

[0003] The current mainstream sampling implementation mainly relies on the general operators provided by PyTorch. The sampling process is mainly as follows: sorting the candidate scores of all candidate words and selecting Top-K; performing a first softmax calculation on all candidate scores to obtain a probability distribution; calculating the prefix sum and filtering the candidate set according to the Top-P threshold; performing a second softmax on the filtered results; applying the Min-P strategy for further filtering; and finally performing a log-softmax calculation to return the sampling result. As can be seen from the above process, the above process needs to perform three global softmax normalization operations on all candidate scores, which will generate a large amount of calculation, and at the same time, the complete candidate score and probability matrix need to be maintained, resulting in frequent memory read and write, and thus the inference efficiency is low in large-scale inference scenarios. SUMMARY

[0004] The embodiments of the present application provide a large language model inference fast sampling method, device, server and medium to solve the technical problem of low inference efficiency in large-scale inference scenarios in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a large language model inference fast sampling method, comprising:

[0006] determining whether the input K value is less than a preset K value threshold, and when the input K value is less than the preset K value threshold, obtaining the candidate scores of Top-K candidate words according to the K value;

[0007] dividing the candidate scores of Top-K candidate words by Temperature to obtain optimized candidate scores of Top-K candidate words;

[0008] writing the optimized candidate score distribution of the Top-K candidate words into GPU shared memory, and using thread bundle level reduction, exchange and reduction to exchange and reduce the optimized probability of the Top-K candidate words.

[0009] In a second aspect, the embodiments of the present application also provide a large language model inference fast sampling device, comprising:

[0010] a judgment module, configured to judge whether the input K value is less than a preset K value threshold, and when the input K value is less than the preset K value threshold, acquire candidate scores of Top-K candidate words according to the K value;

[0011] a division module, configured to divide the candidate scores of the Top-K candidate words by a Temperature to obtain optimized candidate scores of the Top-K candidate words;

[0012] a reduction module, configured to write the optimized candidate scores of the Top-K candidate words into a GPU shared memory, and exchange and reduce optimized probabilities of the Top-K candidate words by using a thread bundle level reduction.

[0013] In a third aspect, the embodiments of the present application also provide a server, comprising:

[0014] one or more processors;

[0015] a storage device, configured to store one or more programs,

[0016] when the one or more programs are executed by the one or more processors, the one or more processors implement any of the large language model inference fast sampling methods provided by the above embodiments.

[0017] In a fourth aspect, the embodiments of the present application also provide a storage medium containing computer executable instructions, which are used to execute the large language model inference fast sampling method provided by the above embodiments when executed by a computer processor.

[0018] The large language model inference fast sampling method, device, server and medium provided by the embodiments of the present application judge whether the input K value is less than a preset K value threshold, and when the input K value is less than the preset K value threshold, acquire candidate scores of Top-K candidate words according to the K value; divide the candidate scores of the Top-K candidate words by a Temperature to obtain optimized candidate scores of the Top-K candidate words; write the optimized candidate scores of the Top-K candidate words into a GPU shared memory, and exchange and reduce optimized probabilities of the Top-K candidate words by using a thread bundle level reduction. By using Top-K to reduce the data amount of the temperature coefficient scaling operation before the temperature coefficient scaling operation, the data amount of the later global softmax normalization is further reduced. And because the data amount involved is significantly reduced compared with before, the communication cost between different threads can be reduced by using the thread bundle level reduction, and the synchronization operation overhead is greatly reduced. Without reducing the inference accuracy, the inference efficiency is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in connection with the following drawings:

[0020] Figure 1 is a flowchart of a large language model inference fast sampling method provided by an embodiment of the application;

[0021] Figure 2 is a flowchart of a large language model inference fast sampling method provided by an embodiment of the application;

[0022] Figure 3 is a structural diagram of a large language model inference fast sampling device provided by an embodiment of the application;

[0023] Figure 4 is a structural diagram of a server provided by an embodiment of the application. DETAILED DESCRIPTION

[0024] The application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the application, rather than limit the application. In addition, it should be noted that only parts related to the application are shown in the drawings, rather than all parts.

[0025] Embodiment 1

[0026] Figure 1 is a flowchart of a large language model inference fast sampling method provided by an embodiment of the application. The embodiment can be applicable to fast sampling of large language model inference. The method can be executed by a large language model inference fast sampling device, and specifically includes the following steps:

[0027] In step 110, it is determined whether the input K value is less than a preset K value threshold. When the input K value is less than the preset K value threshold, the Top-K candidate word candidate score is obtained according to the K value.

[0028] Top-K is a term in the fields of data mining, information retrieval, and machine learning. It means selecting the top K elements with the highest scores from a set of data. In this embodiment, Top-K can be an important parameter in the text generation process of a large language model (LLM), which is used to select candidate words according to the candidate score of each candidate word in the word table output by the model. It is equivalent to discarding a large number of candidate words with the lowest probability according to the probability distribution, and only keeping the K candidate words with the highest probability.

[0029] Exemplarily, the value of K can be input by the user according to requirements. After receiving the value of K input by the user, the value of K can be compared with a preset threshold value of K, which is exemplarily set as 1024. After determining that the value of K is less than the threshold value of K, the candidate scores of the top-K candidate words can be obtained by sorting the candidate words according to the candidate scores. In this way, a large number of useless candidate words can be avoided to occupy the storage space.

[0030] In step 120, the candidate scores of the top-K candidate words are divided by the temperature to obtain the optimized candidate scores of the top-K candidate words.

[0031] The temperature is a parameter for adjusting the original probability distribution of the model output, which controls the degree of “sharpening” or “smoothing” of the output probability distribution by linearly scaling the candidate scores. Sharpening the distribution can amplify the gap between high-probability words and low-probability words, making the output more certain and conservative; smoothing the distribution can reduce the probability difference, giving low-probability words more opportunities to be selected, making the output more random and diverse. Unlike the prior art, the candidate scores of all candidate words are not divided by the temperature, but the candidate scores of the top-K candidate words obtained by the above steps are divided by the temperature, which can effectively reduce the amount of data participating in the division operation.

[0032] In step 130, the optimized candidate score distribution of the top-K candidate words is written into the GPU shared memory, and the optimized probability of the top-K candidate words is exchanged and reduced by using the thread bundle level reduction.

[0033] The optimized candidate score distribution of the top-K candidate words obtained by the above steps is written into the GPU shared memory. The GPU shared memory can be accessed by all threads in a block in GPU programming, which belongs to on-chip memory and has a faster access speed than the access speed of the video memory. Therefore, the optimized candidate scores of the top-K candidate words can be loaded into the GPU shared memory to facilitate fast reduction. Exemplarily, the optimized candidate scores of the top-K candidate words can be subjected to the first softmax, and the result after the softmax is written into the shared memory.

[0034] In addition, due to the use of Top-K candidate words, after softmax, the data is still small, so that the data size is small, and the warp-level reduction is more efficient than the block-level reduction. A thread bundle usually contains 32 threads, which can exchange and share data in their respective registers through a thread bundle shuffle instruction (such as __shfl_down_sync) with almost no synchronization overhead. The existing inter-thread communication needs to write data to shared memory first, synchronize, and then read by other threads, and then exchange and reduce the optimization probability of the Top-K candidate words. As can be seen from the above process, the existing method has a large delay and synchronization overhead,

[0035] In the embodiment, whether the input K value is less than the preset K value threshold is judged. When the K value is less than the preset K value threshold, the candidate scores of Top-K candidate words are obtained according to the K value. The candidate scores of Top-K candidate words are divided by Temperature to obtain the optimized candidate scores of Top-K candidate words. The optimized candidate scores of Top-K candidate words are distributed in the GPU shared memory, and the optimization probability of Top-K candidate words is exchanged and reduced by using warp-level reduction. By using Top-K to reduce the data amount of the temperature coefficient scaling operation before the temperature coefficient scaling operation, the data amount of the global softmax normalization operation in the later stage is further reduced. And because the amount of data involved is significantly reduced compared to before, the communication cost between different threads can be reduced by using warp-level reduction, and the synchronization operation overhead is greatly reduced. Without reducing the inference accuracy, the inference efficiency is greatly improved.

[0036] In a preferred embodiment of the present embodiment, the method can further include the following steps: selecting the maximum value and the sum from the reduced results; when the preset number is greater than 2 and is an exponential value of 2, a reduction binary tree is established for the preset number of threads in the thread bundle, and the sum of the reduction values is obtained by using the calculation of the reduction binary tree; the sum of the reduction values is assigned to the root node of the reduction binary tree, and the value of the root node is decomposed, and the decomposition result is distributed to all threads, so that each thread can calculate the corresponding prefix sum result. In order to realize Top-P (Nucleus Sampling), the prefix sum of the probability needs to be calculated. Top-P sampling will retain the minimum candidate word set whose probability accumulation just exceeds the threshold P (for example, 0.9), and filter out the long-tail distribution with low probability. By using the above method, the exclusive prefix sum corresponding to each thread can be obtained by using the value corresponding to each thread. By skillfully using the tree structure access mode, all calculations can be completed in only a logarithmic number of steps, and high-performance Top-P sampling can be realized.

[0037] Further, the method can further increase the following steps: receiving an incoming Top-P value; sequentially calculating the sum of the prefix sum results of all threads before each thread, stopping decomposition when the sum of the prefix sum results is greater than the incoming P threshold, and determining the minimum candidate word set greater than the threshold P according to the prefix sum reserved probability cumulative sum. Top-p is a commonly used sampling strategy when generating text, which is used to control the diversity and quality of the model output text. From the word with the highest probability, the cumulative probability is accumulated until the cumulative probability reaches a preset threshold p (0 < p ≤ 1), and then only the next word is sampled from the cumulative probability set. Using the prefix sum results recorded in the aforementioned reduction binary tree, the minimum candidate word set satisfying the sum greater than the threshold P is selected. The Top-P strategy can be quickly implemented with the least computing resources.

[0038] Embodiment two

[0039] Figure 2 is a flowchart of a large language model inference fast sampling method provided by embodiment two of the present application. Based on the above-mentioned embodiment, the embodiment is optimized by writing the optimized candidate score distribution of the Top-K candidate words into the GPU shared memory, and using thread bundle level reduction, exchange and reduction of the optimized probability of the Top-K candidate words. The specific optimization is: normalizing the candidate scores of the Top-K candidate words, and calculating the exponential function value of the optimized candidate score of each candidate word in Top-K based on the normalized value, and writing the corresponding distribution of the exponential function value into the GPU shared memory. Correspondingly, the method can further increase the following steps: according to the incoming K value, a corresponding number of threads are allocated according to the K value; the candidate score of the Top-K candidate word is obtained, the candidate score of the Top-K candidate word is divided by Temperature to obtain the optimized candidate score of the Top-K candidate word, and the specific optimization is: each thread obtains the candidate score of the corresponding candidate word, normalizes the candidate score, and calculates the optimized candidate score of each candidate word by dividing the normalized candidate score by Temperature using each thread; the optimized probability of the Top-K candidate word is reduced, exchanged and reduced by thread bundle level, including: according to the exponential function value calculated by each thread, each thread obtains the sum of the exponential function values; each thread obtains the sum of the exponential function values of all threads before itself and itself.

[0040] Referring to Figure 2 , the large language model inference fast sampling method comprises:

[0041] At step 210, according to the incoming K value, a corresponding number of threads is allocated according to the K value, and it is judged whether the incoming K value is less than a preset K value threshold. When the K value is less than the preset K value threshold, each thread obtains a corresponding candidate score of a candidate word, normalizes the candidate score, and calculates an optimized candidate score of each candidate word by dividing the normalized candidate score by Temperature using each thread.

[0042] For example, a thread block can be started, and the number of threads is equal to the K value. Each thread thread_id is responsible for processing a candidate word in the Top-K and its original score. Then, the candidate score of each candidate word is normalized based on the maximum candidate score. The normalization processing can reasonably control the range of the candidate score and avoid overflow in subsequent calculations. Each process calculates the optimized candidate score of each candidate word by dividing the normalized candidate score corresponding to itself by Temperature.

[0043] At step 220, according to the exponential function value calculated by each thread, each thread obtains the sum of the exponential function values, and each thread obtains the sum of the exponential function values of all threads before itself and itself.

[0044] Since a softmax operation is usually required before reduction, in the present embodiment, the above-mentioned thread can perform an exponential function operation on the normalized candidate score of the candidate word corresponding to itself. Because the exponential function grows extremely rapidly when the input is large, it is easy to exceed the maximum value that can be represented by the computer data type (such as float32). By normalization, overflow can be avoided. By additive reduction, the normalization constant of the entire probability distribution is obtained, and each thread obtains the sum of the exponential function values of all threads before itself and itself. Because the data size BLOCk_SIZE is controllable and small, the expensive calculation result can be permanently saved in the fastest register and used at any time without being discarded. For example, the sum of the threadExp of all threads is calculated, that is, the denominator sum(exp(x_i-max)) in the Softmax formula. Each thread can obtain this global sum SumAll. Through the cooperation of all threads, the threadExp values in the respective registers are summed through the efficient BlocKScan / BlocKReduce operation of the CUB library to obtain the global denominator SumAll

[0045] Each thread also obtains the sum of the threadExp of all threads before itself. Assuming that the threads are sorted in descending order of threadExp, then:

[0046] The prefix sum of thread 0 is S0 = exp0

[0047] Prefix sum of thread 1: S1 = exp0 + exp1

[0048] Prefix sum of thread 2: S2 = exp0 + exp1 + exp2 ...

[0049] Prefix sum of thread K-1: S_{K-1} = SumAll.

[0050] At step 230, each thread judges whether the sum of the exponential function values of itself and all threads before itself in the order of the sequence number is greater than the input P threshold value, determines the thread with the minimum sequence number greater than the input P threshold value, and determines the minimum candidate word set according to the thread with the minimum sequence number.

[0051] Because the prefix sum is increasing, that is, the threads are arranged in descending order of probability, all threads before the first thread that makes the cumulative probability exceed p * SumAll can be selected from top to bottom, and other threads will be filtered out. The corresponding prefix sum of each thread can be generated almost without additional cost.

[0052] At step 240, the probability of each candidate word in the minimum candidate word set is obtained, candidate words smaller than the minimum probability in the minimum candidate word set are removed according to the corresponding minimum probability in the input Min-P, to obtain an optimal candidate word set, and the softmax probability and the logarithm of the candidate word in the optimal candidate word set are written into the global memory.

[0053] Generally, the Min-P operation can also be performed again to set a minimum probability threshold, retain words with a probability ≥ Min-P, and then re-normalize sampling.

[0054] In this embodiment, Min-P is actually equivalent to exp(x_i - max) > min_p, and the exp(x_i - max) obtained by the above steps is used until it is greater than top-p*sum(exp(x_i - max)). In this way, the amount of calculation can be further reduced.

[0055] In this embodiment, the optimized candidate score distribution of the Top-K candidate words is written into the GPU shared memory, and the optimized probability of the Top-K candidate words is exchanged and reduced by using the thread bundle level reduction. Specifically, the candidate scores of the Top-K candidate words are normalized, and the exponential function value of the optimized candidate score of each candidate word in the Top-K is calculated based on the normalized value, and the exponential function value corresponding distribution is written into the GPU shared memory. Correspondingly, the method can further include the following steps: according to the input K value, a corresponding number of threads are allocated according to the K value; the candidate score of the Top-K candidate word is obtained, the candidate score of the Top-K candidate word is divided by Temperature, and the optimized candidate score of the Top-K candidate word is obtained. Specifically, each thread obtains the candidate score of the corresponding candidate word, normalizes the candidate score, and calculates the optimized candidate score of each candidate word by dividing the normalized candidate score by Temperature using each thread; the optimized probability of the Top-K candidate words is exchanged and reduced by using the thread bundle level reduction, including: according to the exponential function value calculated by each thread, each thread obtains the sum of the exponential function values; each thread obtains the sum of the exponential function values of all threads before itself and itself. Single calculation and permanent storage of molecules can be realized, and the denominator can be obtained by reduction summation, and the stored molecules can be directly used for calculation, which can reduce the number of exp() calculation by half, and the intermediate results are temporarily stored in the high-speed register, further improving the inference efficiency.

[0056] Embodiment three

[0057] Figure 3 is a structural schematic diagram of the large language model inference fast sampling device provided by Embodiment Three of the present application, referring to Figure 3 , the large language model inference fast sampling device comprises:

[0058] The judgment module 310 is used for judging whether the input K value is less than the preset K value threshold. When it is less than the preset K value threshold, the candidate score of the Top-K candidate word is obtained according to the K value;

[0059] The division module 320 is used for dividing the candidate score of the Top-K candidate word by Temperature to obtain the optimized candidate score of the Top-K candidate word.

[0060] The reduction module 330 is used for writing the optimized candidate score distribution of the Top-K candidate words into the GPU shared memory, and exchanging and reducing the optimized probability of the Top-K candidate words by using the thread bundle level reduction.

[0061] The large language model inference fast sampling device provided by the embodiment, by judging whether the input K value is less than the preset K value threshold, when less than the preset K value threshold, according to the K value, the candidate score of Top-K candidate word is obtained; the candidate score of Top-K candidate word is divided by Temperature, and the optimized candidate score of Top-K candidate word is obtained; the optimized candidate score of Top-K candidate word is written into the GPU shared memory, and the optimized probability of Top-K candidate word is exchanged and reduced by using the thread bundle level reduction. By using Top-K to reduce the data amount of temperature coefficient scaling operation before temperature coefficient scaling operation, the data amount of later global softmax normalization is further reduced. And because the data amount is significantly reduced compared with before, the communication cost between different threads can be reduced by using thread bundle level reduction, and the synchronization operation overhead is greatly reduced. Without reducing the inference accuracy, the inference efficiency is greatly improved.

[0062] On the basis of the above embodiments, the reduction module comprises:

[0063] The calculation unit is configured to normalize the candidate score of Top-K candidate word, calculate the exponential function value of the optimized candidate score of each candidate word in Top-K based on the normalized value, and write the corresponding distribution of the exponential function value into the GPU shared memory.

[0064] On the basis of the above embodiments, the device further comprises:

[0065] The selection module is configured to select the maximum value and the total sum from the reduced result.

[0066] The establishment module is configured to, when the preset number is greater than 2 and is an exponential value of 2, establish a reduction binary tree for the preset number of threads in the thread bundle, and obtain the total sum of reduction values by using the calculation of the reduction binary tree.

[0067] The assignment module is configured to assign the total sum of reduction values to the root node of the reduction binary tree, decompose the value of the root node, and distribute the decomposition result to all threads, so that each thread can calculate the corresponding prefix sum result.

[0068] On the basis of the above embodiments, the device further comprises:

[0069] The thread distribution module is configured to distribute a corresponding number of threads according to the input K value according to the K value.

[0070] Correspondingly, the division module comprises:

[0071] In addition to the dividing unit, each thread obtains a candidate score of a corresponding candidate word, normalizes the candidate score, and calculates an optimized candidate score of each candidate word by dividing the normalized candidate score by Temperature using each thread;

[0072] The reduction module comprises:

[0073] The first obtaining unit is configured to obtain a sum of the exponential function values of each thread.

[0074] The second obtaining unit is configured to obtain, using each thread, a sum of the exponential function values of all threads before the thread itself and the thread itself.

[0075] On the basis of the above embodiments, the apparatus further comprises:

[0076] The Top-P value receiving module is configured to receive an incoming Top-P value.

[0077] The set determining module is configured to sequentially calculate a sum of prefix sum results of all threads before each thread, stop decomposition when the sum of the prefix sum results is greater than the incoming P threshold value, and determine a minimum candidate word set greater than the threshold value P according to a prefix sum retention probability accumulation.

[0078] On the basis of the above embodiments, the set determining module comprises:

[0079] The thread determining unit is configured to determine, using each thread, whether a sum of the exponential function values of all threads before the thread itself and the thread itself is greater than the incoming P threshold value, and determine a thread with a minimum sequence number greater than the incoming P threshold value.

[0080] The minimum candidate word set determining unit is configured to determine a minimum candidate word set according to the thread with the minimum sequence number.

[0081] On the basis of the above embodiments, the apparatus further comprises:

[0082] The probability obtaining module is configured to obtain a probability of each candidate word in the minimum candidate word set.

[0083] The removing module is configured to remove, according to a corresponding minimum probability in the incoming Min-P, a candidate word smaller than the minimum probability in the minimum candidate word set, to obtain an optimal candidate word set.

[0084] The writing module is configured to write a softmax probability and a logarithm of a candidate word in the optimal candidate word set into a global memory.

[0085] The large language model reasoning fast sampling device provided by the embodiment of the present application can execute the large language model reasoning fast sampling method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0086] Embodiment four

[0087] Figure 4 A structural schematic diagram of a device provided for the fourth embodiment of the present application. Figure 4 A block diagram of an exemplary device 12 suitable for implementing an embodiment of the present application is shown. Figure 4 The device 12 shown is merely one example and should not be taken as limiting the scope of the embodiments of the present application.

[0088] As Figure 4 shown, the device 12 is in the form of a general-purpose computing device. The components of the device 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

[0089] The bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0090] The device 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by the device 12 and includes both volatile and non-volatile media, removable and non-removable media.

[0091] The system memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache 32. The device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 4 (not shown), commonly referred to as a "hard disk drive"). Although not specifically shown, the device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media including, but not limited to, a magnetic floppy disk drive with a floppy disk, a magnetic hard disk drive, an optical disk drive, a Blu-ray drive, a flash memory drive (e.g., card, stick, key drive) or other storage technology, a tape drive, a solid state drive, a solid state memory drive, a RAID storage system, a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), or other like computer system storage media. Figure 4Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0092] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0093] Device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with device 12, and / or with any device that enables device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0094] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the fast sampling method for large language model inference provided in the embodiments of the present invention.

[0095] Example 5

[0096] Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the fast sampling method for large language model inference as described in any of the above embodiments.

[0097] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0098] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which computer readable program code is embodied. Such propagated data signals can take a wide variety of forms, including but not limited to electro-magnetic signals, optical signals, or any suitable combination thereof. Computer readable signal medium can also be any computer readable medium that is not a storage medium, that is capable of storing the program for use by or in connection with the instruction execution system, apparatus, or device.

[0099] The program code embodied on the computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0100] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments of the present application, electronic mail (email) can be utilized as the distrusting mechanism to effectuate exchange of information.

[0101] Note that the above merely describes preferred embodiments of the present application and the principles of the technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, modifications and substitutions can be made without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A large language model inference fast sampling method, characterized in that, The method comprises: determining whether the input K value is less than a preset K value threshold, and when the input K value is less than the preset K value threshold, obtaining candidate scores of Top-K candidate words according to the K value; dividing the candidate scores of the Top-K candidate words by a Temperature to obtain optimized candidate scores of the Top-K candidate words; writing the optimized candidate score distribution of the Top-K candidate words into a GPU shared memory, and performing thread bundle level reduction, exchange and reduction on the optimized probability of the Top-K candidate words; selecting a maximum value and a total sum from the reduced results; when the preset number is greater than 2 and is an exponential value of 2, establishing a reduction binary tree for the preset number of threads in the thread bundle, and obtaining a total sum of reduction values by using the reduction binary tree for calculation; assigning the total sum of the reduction values to a root node of the reduction binary tree, decomposing the value of the root node, and distributing the decomposition results to all threads, so that each thread can calculate a corresponding prefix sum result.

2. The method of claim 1, wherein, The method further comprises: normalizing the candidate scores of the Top-K candidate words, calculating an exponential function value of the optimized candidate score of each candidate word in the Top-K based on the normalized value, and writing the corresponding distribution of the exponential function value into the GPU shared memory.

3. The method of claim 2, wherein, The method further comprises: allocating a corresponding number of threads according to the input K value according to the K value; Correspondingly, the method of obtaining the candidate scores of the Top-K candidate words, dividing the candidate scores of the Top-K candidate words by the Temperature to obtain the optimized candidate scores of the Top-K candidate words comprises: each thread obtains the candidate score of the corresponding candidate word, normalizes the candidate score, and calculates the optimized candidate score of each candidate word by dividing the normalized candidate score by the Temperature using each thread; The method further comprises: each thread obtains the sum of the exponential function values of all threads before itself and itself according to the exponential function value calculated by each thread.

4. The method of claim 1, further comprising: receiving an input Top-P value; calculating the sum of the prefix sum results of all threads before each thread in turn, stopping decomposition when the sum of the prefix sum results is greater than the input P threshold, and determining a minimum candidate word set greater than the threshold P according to the prefix sum and the cumulative sum of the retention probability. The method further comprises:

5. The method of claim 4, wherein, each thread determines whether the sum of the exponential function values of all threads before itself and itself is greater than the input P threshold, and determines the thread with the minimum sequence number greater than the input P threshold; determining a minimum candidate word set according to the thread with the minimum sequence number. The method further comprises:

6. The method of claim 4, wherein, ​ obtain a probability of each candidate word in the minimum candidate word set; remove candidate words in the minimum candidate word set that are less than the minimum probability according to the minimum probability corresponding to the Min-P, to obtain an optimal candidate word set; write the softmax probability and the logarithm of the candidate words in the optimal candidate word set into the global memory.

7. A large language model inference fast sampling apparatus, characterized by comprising: comprise: a judgment module configured to judge whether the incoming K value is less than a preset K value threshold, and when the K value is less than the preset K value threshold, obtain candidate scores of Top-K candidate words according to the K value; a division module configured to divide the candidate scores of the Top-K candidate words by a Temperature to obtain optimized candidate scores of the Top-K candidate words; a reduction module configured to write the optimized probability distribution of the Top-K candidate words into a GPU shared memory, and exchange and reduce the optimized probability of the Top-K candidate words by using a thread bundle level reduction; a selection module configured to select a maximum value and a total sum from the results after reduction according to the results after reduction; an establishment module configured to, when a preset number is greater than 2 and is an exponential value of 2, establish a reduction binary tree for the preset number of threads in the thread bundle, and obtain a reduction value total sum by using calculation of the reduction binary tree; an assignment module configured to assign the reduction value total sum to a root node of the reduction binary tree, decompose the value of the root node, and distribute the decomposition results to all threads, so that each thread can calculate a corresponding prefix sum result.

8. A server, characterized by comprise: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the large language model inference fast sampling method according to any one of claims 1-6.

9. A storage medium containing computer-executable instructions, wherein: the computer executable instructions, when executed by a computer processor, are used to perform the large language model inference fast sampling method according to any one of claims 1-6.

Images