Method and system for distributing operator fusion mechanism of vector quantization large language model

Abstract

The invention provides an operator fusion mechanism distribution method and system for a vector quantization large language model, and the method comprises the following steps: obtaining layout data which comprises a dequantization data layout and a layout needed by a calculation kernel; calculating the number of data shuffling times based on the layout data, comparing the number of data shuffling times with a preset threshold value to dynamically respond to a decision mechanism, and when the number of data shuffling times is smaller than the preset threshold value, adopting a first decision mechanism which at least comprises a register level operator fusion mechanism; and when the number of data shuffling times is greater than or equal to a preset threshold value, adopting a second decision mechanism which at least comprises a shared memory level operator fusion mechanism. According to the operator fusion mechanism allocation method and system for the vector quantization large language model, the problems of bandwidth bottleneck and delay caused by frequent transmission of data between a shared memory and a register are solved, and therefore more efficient dequantization and calculation fusion are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method and system for allocating an operator fusion mechanism of a vector quantized large language model. Background Art

[0002] At present, the fusion of de-vectorization and subsequent operators is mainly achieved by using the shared memory of the GPU to exchange and store data within the thread block, avoiding frequent access to the global memory and improving data transmission efficiency. Shared memory acts as an intermediate cache, allowing threads to quickly exchange data without having to frequently access the global memory. This storage method takes advantage of the low latency of shared memory and greatly reduces the memory bandwidth usage. After storing the sub-vector in the shared memory, each thread will extract the required specific data from the shared memory according to the needs of the computing task.

[0003] The existing shared memory level fusion scheme uses shared memory as an intermediary to adjust the dequantized data layout to adapt to computing requirements. Although this method solves the layout mismatch problem, it also causes a new bottleneck: frequent data transfers are carried out between shared memory and registers. Each data transfer increases the communication volume, resulting in frequent and inefficient transfers between registers and shared memory, forming a significant performance bottleneck. Summary of the invention

[0004] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a method and system for allocating an operator fusion mechanism for a vector quantized large language model, so as to solve the problem of frequent arrival and low efficiency of communication transmission in the above-mentioned shared memory level fusion scheme.

[0005] In a first aspect, the present invention provides a method for allocating an operator fusion mechanism for a vector quantized large language model, the method comprising the following steps:

[0006] Acquire layout data, wherein the layout data includes a dequantized data layout and a layout required for a computing kernel;

[0007] The number of data shuffles is calculated based on the layout data, and the number of data shuffles is compared with a preset threshold value to dynamically respond to the decision mechanism, wherein:

[0008] When the number of data shuffles is less than the preset threshold, a first decision mechanism is adopted, wherein the first decision mechanism at least includes a register-level operator fusion mechanism;

[0009] When the number of data shuffles is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least includes a shared memory level operator fusion mechanism.

[0010] In a possible implementation of the present application, the obtaining of layout data specifically includes:

[0011] Obtaining the dequantized data layout based on the size of the sub-vector in the vector quantization;

[0012] The required layout of the computing kernel is obtained based on the specification of the current computing operator.

[0013] In a possible implementation of the present application, the calculating the number of data shuffling times based on the layout data specifically includes:

[0014] Calculating a size of a thread warp based on the layout data, wherein the size of the thread warp is obtained based on a ratio of the dequantized data layout to a layout required by the computing kernel;

[0015] The number of data shuffles is obtained by calculation based on the size of the thread warp, wherein the number of data shuffles is obtained by subtracting one from the number of the thread warp.

[0016] In a possible implementation of the present application, the dynamic response decision mechanism specifically includes:

[0017] Extract the data shuffle times N and the preset threshold N t , wherein the preset threshold N t =5;

[0018] Calculate the data shuffling times N and the preset threshold N t The size of

[0019] When N<5, the register-level operator fusion mechanism is used to perform operator calculation;

[0020] When N≥5, the shared memory level operator fusion mechanism is used to perform operator calculation.

[0021] In a possible implementation of the present application, operator calculation is performed based on a register-level operator fusion mechanism, specifically including:

[0022] The grouping rules of thread mapping are obtained by performing offline processing based on the dequantized data layout and the layout required by the computing kernel;

[0023] Dequantizing the quantized data using the grouping rule of thread mapping to obtain the dequantized data;

[0024] Based on the dequantized data combined with the grouping rule, the register exchange between threads is completed;

[0025] After completing the inter-thread register exchange, the calculation of the calculation operator is performed.

[0026] In a possible implementation of the present application, the grouping rules for thread mapping obtained by performing offline processing based on the dequantized data layout and the layout required by the computing kernel specifically include:

[0027] Extracting the dequantized data and initializing the computing thread;

[0028] The computing threads are grouped according to the required dequantized data to obtain mini-thread bundles;

[0029] A grouping rule for thread mapping is obtained based on the mini-warps and their corresponding dequantized data, wherein the grouping rule includes a shuffle offset.

[0030] In a possible implementation of the present application, completing the inter-thread register exchange based on the dequantized data combined with the grouping rule specifically includes:

[0031] The dequantized data in different threads are shuffled according to the grouping rule to complete the inter-thread register exchange, wherein:

[0032] When shuffling data, a preset interface formula is used for shuffling, and the interface formula is as follows:

[0033] Register ← shuffle xor (register, shuffle offset);

[0034] Specifically, during the exchange, the original register value of the thread is exchanged with the register value of the corresponding shuffle offset.

[0035] In a second aspect, the present invention provides an operator fusion mechanism allocation system for a vector quantized large language model, the system comprising:

[0036] An acquisition module, used for acquiring layout data, wherein the layout data includes a dequantized data layout and a layout required for a computing kernel;

[0037] A comparison module, configured to calculate the number of data shuffles based on the layout data, and compare the number of data shuffles with a preset threshold value to dynamically respond to a decision mechanism;

[0038] A module is used to adopt a first decision mechanism when the number of data shuffles is less than the preset threshold, and the first decision mechanism at least includes a register-level operator fusion mechanism; and when the number of data shuffles is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least includes a shared memory-level operator fusion mechanism.

[0039] In a third aspect, the present invention provides an electronic device, the electronic device comprising: a processor and a memory;

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned operator fusion mechanism allocation method for the vector quantization large language model.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned method for allocating an operator fusion mechanism for a vector quantized large language model.

[0043] As described above, the method and system for allocating the operator fusion mechanism of the vector quantization large language model of the present invention have the following beneficial effects:

[0044] (1) Register-level fusion can directly perform shuffle operations between mini-warps, requiring only one data transfer, greatly reducing operation latency;

[0045] (2) Register-level fusion reduces the need for synchronization instructions through direct register exchange, further improving efficiency;

[0046] (3) The hierarchical adaptive operator fusion allocation can maximize the utilization of hardware resources and avoid the performance bottleneck caused by fixed fusion strategies, making it highly versatile when processing different vector quantization schemes and different types of operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Shown is a schematic diagram of a scene of an electronic device in an embodiment of the present invention;

[0048] Figure 2 Shown is a flow chart of an operator fusion mechanism allocation method for a vector quantized large language model in one embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram showing the steps of a dynamic response decision mechanism of an operator fusion mechanism allocation method of a vector quantized large language model of the present invention in one embodiment;

[0050] Figure 4 It is a schematic diagram showing the steps corresponding to the fusion mechanism of the first decision mechanism and the second decision mechanism in an embodiment of the operator fusion mechanism allocation method of the vector quantization large language model of the present invention;

[0051] Figure 5 An example diagram showing data exchange within a warp based on data shuffling in an embodiment of an operator fusion mechanism allocation method for a vector quantized large language model of the present invention;

[0052] Figure 6 Shown is a schematic structural diagram of an operator fusion mechanism allocation system for a vector quantized large language model of the present invention in one embodiment;

[0053] Figure 7 It is a schematic structural diagram of an electronic device of the present invention in one embodiment. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0055] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0056] The following embodiments of the present invention provide a method for allocating an operator fusion mechanism for a vector quantized large language model, which can be applied to: Figure 1 The electronic device shown. The electronic device described in the present invention may include a mobile phone 11 with a wireless charging function, a tablet computer 12, a laptop computer 13, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present invention does not impose any restrictions on the specific type of the electronic device.

[0057] For example, the electronic device may be a station (STATION, ST) in a WLAN with a wireless charging function, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with a wireless charging function, a computing device or other processing device, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future evolved Public Land Mobile Network (PLMN) or a mobile terminal in a future evolved Non-terrestrial Network (NTN), etc.

[0058] With the rapid development of artificial intelligence, large language models (such as GPT, BERT, etc.) have shown great potential in natural language processing, machine translation, and dialogue systems. However, as the model scales up and the number of parameters reaches billions or even hundreds of billions, the demand for computing resources and memory also increases dramatically, which puts tremendous pressure on the hardware, especially during the inference process, where frequent data loading and processing leads to increased computing latency. Therefore, how to reduce memory usage and accelerate inference has become a key challenge in deploying large models.

[0059] In order to solve the resource demand challenges brought by large models, quantization technology has become one of the important means to optimize computing and storage resources. Quantization technology significantly reduces the storage requirements of the model by converting high-precision floating-point numbers (such as FP16 or FP32) in the model into low-precision representations (such as INT8, INT4). It is worth noting that although this element-by-element quantization method is relatively simple, it can only achieve a very limited compression rate because it does not consider the correlation information between elements. Among them, when the quantization bit width is further reduced, the model accuracy is prone to a significant decrease. In order to overcome the shortcomings of element-by-element quantization, vector quantization technology came into being. Vector quantization can effectively solve the shortcomings of element-by-element quantization in compression rate, quantization error and structural information capture, and has shown significant potential in the inference acceleration of large language models.

[0060] Different from element-by-element quantization, vector quantization maps a set of high-dimensional vectors to a limited set of discrete codewords, that is, the data representation is compressed through the codebook. Specifically, the vector quantization technology first generates a codebook containing representative vectors through a clustering algorithm. During the quantization process, it replaces each vector with its closest codebook index. When dequantizing, the corresponding codeword is searched from the codebook through the index to restore the approximate original data. The technical solution in the embodiment of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiment of the present invention.

[0061] Vector quantization is mainly used to compress weights and KV cache (Key-Value Cache) in large models. Weights are the part of model parameters that takes up a lot of storage space, while KV cache is used to store intermediate calculation results during model inference. These are all heavily dependent on memory. However, although vector quantization performs well in compression, how to convert it into an increase in inference speed is a challenge. In the inference process, compressed data must first be dequantized before participating in the calculation, that is, the index is restored to the original vector by searching the codebook. This process often needs to be performed at each stage of inference, because the quantized data cannot directly participate in matrix multiplication or other operations. Therefore, optimizing the fusion kernel and integrating the dequantization and calculation process more efficiently can not only reduce memory access latency, but also significantly improve data processing speed.

[0062] At present, the fusion of de-vectorization and subsequent operators is mainly achieved by using the shared memory of the GPU to exchange and store data within the thread block. Specifically, the bandwidth of the shared memory is limited. When faced with high-frequency data exchange, it cannot support the data flow requirements in large-scale parallel computing. Frequent data transmission consumes precious bandwidth resources, further weakening the computing efficiency. In addition, the access speed of the shared memory is much lower than that of the register, resulting in delays every time data is transferred from the shared memory to the register for calculation. These delays continue to accumulate in large-scale parallel computing, resulting in the failure to fully utilize computing resources and reducing the overall performance of the system. At the same time, the capacity of the shared memory is limited. Especially when processing high-dimensional vectors and large-scale data, the storage and transmission of the de-quantization results are not only limited by the bandwidth of the shared memory, but also by its storage capacity. The shared memory cannot accommodate all the de-quantized data at one time, which means that the data must be loaded and transmitted in batches, further increasing the delay and complexity of the system.

[0063] To this end, the present invention introduces a hierarchical operator fusion scheme, expands the register level fusion based on the shared memory level fusion, and can dynamically select the optimal scheme according to the overhead of the two schemes, solving the bandwidth bottleneck and delay problems caused by the frequent transmission of data between shared memory and registers, thereby achieving more efficient dequantization and computational fusion.

[0064] Specifically, see Figure 2 In one embodiment of the invention, the operator fusion mechanism allocation method of the vector quantization large language model of the present invention comprises the following steps:

[0065] Step S202, obtaining layout data, wherein the layout data includes a dequantized data layout and a layout required for a computing kernel;

[0066] Step S204, calculating the number of data shuffling times based on the layout data, and comparing the number of data shuffling times with a preset threshold value to dynamically respond to a decision mechanism;

[0067] Step S206, when the number of data shuffling times is less than the preset threshold, a first decision mechanism is adopted, wherein the first decision mechanism at least includes a register-level operator fusion mechanism;

[0068] Step S208: When the number of data shuffling times is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least includes a shared memory level operator fusion mechanism.

[0069] It should be noted that the present invention provides an extended register-level fusion solution, which effectively bypasses the frequent transmission delays caused by shared memory computing in certain application scenarios through register-level data exchange, realizes the data layout conversion from dequantization to operator calculation, improves the fusion efficiency, and realizes an adaptive dynamic decision-making mechanism, which automatically selects shared memory level fusion or register level fusion according to the dequantization data layout and the layout required by the computing kernel.

[0070] Specifically, in this embodiment, the layout data is first obtained. Accordingly, the obtaining of the layout data specifically includes: obtaining the dequantization data layout based on the size of the sub-vector in the vector quantization, and obtaining the required layout of the computing kernel based on the specification of the current computing operator, wherein the size of the sub-vector and the specification of the computing operator are known given quantities.

[0071] Furthermore, after obtaining the layout data, the number of data shuffles can be obtained by calculation based on the layout data. The subsequent specification will provide a detailed explanation. After obtaining the number of data shuffles, it can be compared with a preset threshold to perform a dynamic response decision mechanism. The greater the difference between the dequantized data layout and the layout required by the computing kernel, the greater the demand for data shuffling. When the number of required data exchanges exceeds a certain threshold, the overhead will be higher than the data exchange at the shared memory level. Therefore, it is necessary to calculate the number of data shuffles based on the layout data to adaptively adjust the level of operator fusion.

[0072] Specifically, experiments have shown that the access latency of shared memory is almost five times that of the combination of register access and shuffle operations. Therefore, for quantized tensors that require less than 5 shuffle operations, a register-level fusion method can be used. For tensors that require more shuffle operations, the traditional shared memory-level fusion scheme is maintained. Through adaptive decision-making, the optimal fusion method is effectively selected according to the characteristics of different quantized tensors, which can optimize system efficiency while ensuring performance.

[0073] Furthermore, if Figure 3 As shown, it is a schematic diagram of the steps of the dynamic response decision-making mechanism, which specifically includes the following steps:

[0074] Step 1: Extract the data shuffling times N and the preset threshold N t , wherein the preset threshold N t =5;

[0075] Step 2: Calculate the data shuffling times N and the preset threshold N t The size of which enters the judgment branch, specifically,

[0076] When N<5, the first decision mechanism is adopted, that is, the register-level operator fusion mechanism is adopted to perform operator calculation;

[0077] When N≥5, the second decision mechanism is sampled, that is, the shared memory level operator fusion mechanism is used to perform operator calculation.

[0078] Specifically, Figure 4 As shown, it is a schematic diagram of the steps corresponding to the fusion mechanism of the first decision mechanism and the second decision mechanism, wherein: Figure 4 The right-hand branch corresponds to the steps of the register-level operator fusion mechanism shown in the first decision mechanism. Figure 4 The left branch corresponds to the steps of the shared memory level operator fusion mechanism indicated by the second decision mechanism.

[0079] Further, in an embodiment of the invention, the number of data shuffling times calculated based on the layout data specifically includes:

[0080] Calculating a size of a thread warp based on the layout data, wherein the size of the thread warp is obtained based on a ratio of the dequantized data layout to a layout required by the computing kernel;

[0081] The number of data shuffles is obtained by calculation based on the size of the thread warp, wherein the number of data shuffles is obtained by subtracting one from the number of the thread warp.

[0082] It should be noted that, in this embodiment, Figure 5For example, an example diagram of data exchange within a thread bundle based on data shuffling is shown, wherein the quantized data is a "16×2" matrix, which needs to be dequantized to obtain the original "16×16" layout. Therefore, the dequantized data layout is "8", indicating that the register of a thread will hold "8" consecutive dequantized data, and the corresponding computing kernel requires a layout of "2", that is, each thread only needs to hold "2" consecutive elements to complete the calculation. Therefore, based on the ratio of the dequantized data layout to the layout required by the computing kernel, the thread bundle is obtained, and the size of the thread bundle is obtained as "4", and the number of shuffles is the number of the thread bundle minus one. Therefore, for Figure 5 For the example scenario, the number of shuffling times corresponds to "3".

[0083] Furthermore, in one embodiment of the invention, operator calculation is performed based on a register-level operator fusion mechanism, specifically including:

[0084] The grouping rules of thread mapping are obtained by performing offline processing based on the dequantized data layout and the layout required by the computing kernel;

[0085] Dequantizing the quantized data using the grouping rule of thread mapping to obtain the dequantized data;

[0086] Based on the dequantized data combined with the grouping rule, the register exchange between threads is completed;

[0087] After completing the inter-thread register exchange, the calculation of the calculation operator is performed.

[0088] It should be noted that, in this embodiment,

[0089] The thread mapping grouping rule obtained by performing offline processing based on the dequantized data layout and the required layout of the computing kernel specifically includes: extracting the dequantized data, initializing the computing threads, and grouping the computing threads according to the required dequantized data to obtain mini-thread bundles, wherein the mini-thread bundles correspond to the dequantized data required to be processed after grouping, and then obtaining the thread mapping grouping rule based on the mini-thread bundles and their corresponding dequantized data, wherein the grouping rule includes a shuffle offset, wherein, for example, Figure 5 As shown, the thread program numbers in the mini-thread bundle include "sequence number 0, sequence number 1, sequence number 2 and sequence number 3", the color block corresponds to the dequantized data to be processed, and then the grouping rule is obtained based on the mini-thread bundle and the corresponding dequantized data, which specifically corresponds to the offset rule between different thread program numbers. Specifically, the grouping rule includes a shuffle offset.

[0090] Further, completing the inter-thread register exchange based on the dequantized data combined with the grouping rule specifically includes:

[0091] The dequantized data in different threads are shuffled according to the grouping rule to complete the inter-thread register exchange, wherein the data is shuffled using a preset interface formula, and the interface formula is as follows:

[0092] Register ← shuffle xor (register, shuffle offset);

[0093] Specifically, during the exchange, the original register value of the thread is exchanged with the register value of the corresponding shuffle offset.

[0094] Specifically, shuffle means shuffle, xor means exclusive or operation, and the exchange operation is processed offline in the register, which can realize data exchange in the register device without transferring through shared memory, thereby reducing system latency. Specifically, Figure 5 As shown, for a warp of "4", "3" shuffle operations need to be performed, and each shuffle is performed based on the corresponding shuffle offset. The process is as follows:

[0095]

[0096] Among them, Tid X.[Y] represents the Yth element of the register array in the thread with thread number X. For example, Tid0.[1] represents the element with register array index "1" in the thread with "serial number 0". After "3" times of shuffling, it is ensured that the data required by the thread of each serial number during calculation can be dequantized in the thread bundle in which it is located without relying on the data of other thread bundles. In this way, the entire data processing process is optimized, the redundant data exchange between threads is reduced, and the overall computing efficiency is improved.

[0097] The example illustrated in the above embodiment corresponds to the first decision mechanism, that is, it corresponds to the register-level operator fusion mechanism, and when N≥5, the second decision mechanism is sampled, that is, the shared memory-level operator fusion mechanism is used to perform operator calculation, wherein, when shared memory is used for data exchange and storage, for the scenario where the dequantized data layout is "8", each sub-vector is represented by a quantization index, and in the dequantization process, the "32" parallel threads in the thread block are used to read the "8"-dimensional sub-vector from the codebook through the quantization index, and then the dequantized sub-vectors are exchanged between different threads. Since the shared memory-level operator fusion mechanism is a technical means that can be directly applied by technical personnel in this field, it will not be elaborated here.

[0098] An embodiment of the present application also provides an operator fusion mechanism allocation system for a vector quantized large language model. The operator fusion mechanism allocation system for a vector quantized large language model can implement the operator fusion mechanism allocation method for a vector quantized large language model described in the present application. However, the implementation device of the operator fusion mechanism allocation method for a vector quantized large language model described in the present application includes but is not limited to the structure of the operator fusion mechanism allocation system for a vector quantized large language model listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0099] See also Figure 6 In one embodiment, this embodiment provides a vector quantization large language model operator fusion mechanism allocation system 60, the system comprising:

[0100] An acquisition module 61 is used to acquire layout data, wherein the layout data includes a dequantized data layout and a layout required for a computing kernel;

[0101] A comparison module 62, configured to calculate the number of data shuffling times based on the layout data, and compare the number of data shuffling times with a preset threshold value to dynamically respond to a decision mechanism;

[0102] Module 63 is used to adopt a first decision mechanism when the number of data shuffling times is less than the preset threshold, and the first decision mechanism at least includes a register-level operator fusion mechanism; and when the number of data shuffling times is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least includes a shared memory-level operator fusion mechanism.

[0103] Since the specific implementation of this embodiment corresponds to the aforementioned method embodiment, the same details will not be repeated here, and those skilled in the art should also understand that Figure 6 The division of the modules in the embodiment is merely a division of logical functions, and in actual implementation, all or part of them can be integrated into one or more physical entities, and these modules can be implemented entirely in the form of software called by processing elements, or entirely in the form of hardware, or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware.

[0104] In the several embodiments provided by the present invention, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.

[0105] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0106] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0107] The embodiment of the present invention also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0108] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory.

[0109] The memory is used to store computer programs.

[0110] The memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0111] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned operator fusion mechanism allocation method for the vector quantization large language model.

[0112] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0113] like Figure 7As shown, the electronic device of the present invention is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors or processing units 71, a memory 72, and a bus 73 connecting different system components (including the memory 72 and the processing unit 71).

[0114] Bus 73 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0115] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0116] The memory 72 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 721 and / or cache memory 722. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 723 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 73 via one or more data medium interfaces. The memory 72 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0117] A program / utility 724 having a set (at least one) of program modules 7241 may be stored, for example, in the memory 72, such program modules 7241 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 7241 generally perform the functions and / or methods of the embodiments described herein.

[0118] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 74. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 75. Figure 7 As shown, the network adapter 75 communicates with other modules of the electronic device via the bus 73. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for allocating operator fusion mechanisms for a vector quantized large language model, characterized in that: include: Acquire layout data, wherein the layout data includes a dequantized data layout and a layout required for a computing kernel; The number of data shuffles is calculated based on the layout data, and the number of data shuffles is compared with a preset threshold value to dynamically respond to the decision mechanism, wherein: When the number of data shuffles is less than the preset threshold, a first decision mechanism is adopted, wherein the first decision mechanism at least includes a register-level operator fusion mechanism; When the number of data shuffles is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least includes a shared memory level operator fusion mechanism.

2. The method for allocating operator fusion mechanisms for vector quantized large language models according to claim 1, characterized in that: The obtaining of layout data specifically includes: Obtaining the dequantized data layout based on the size of the sub-vector in the vector quantization; The required layout of the computing kernel is obtained based on the specification of the current computing operator.

3. The method for allocating operator fusion mechanisms for vector quantized large language models according to claim 2, characterized in that: The calculating the number of data shuffling times based on the layout data specifically includes: Calculating a size of a thread warp based on the layout data, wherein the size of the thread warp is obtained based on a ratio of the dequantized data layout to a layout required by the computing kernel; The number of data shuffles is obtained by calculation based on the size of the thread warp, wherein the number of data shuffles is obtained by subtracting one from the number of the thread warp.

4. The method for allocating operator fusion mechanisms for vector quantized large language models according to claim 3, characterized in that: The dynamic response decision-making mechanism specifically includes: Extract the data shuffle times N and the preset threshold N t , wherein the preset threshold N t =5; Calculate the data shuffling times N and the preset threshold N t The size of When N<5, the register-level operator fusion mechanism is used to perform operator calculation; When N≥5, the shared memory level operator fusion mechanism is used to perform operator calculation.

5. The method for allocating operator fusion mechanisms for vector quantized large language models according to claim 4, characterized in that: Operator calculation is performed based on the register-level operator fusion mechanism, including: The grouping rules of thread mapping are obtained by performing offline processing based on the dequantized data layout and the layout required by the computing kernel; Dequantizing the quantized data using the grouping rule of thread mapping to obtain the dequantized data; Based on the dequantized data combined with the grouping rule, the register exchange between threads is completed; After completing the inter-thread register exchange, the calculation of the calculation operator is performed.

6. The method for allocating operator fusion mechanisms for vector quantized large language models according to claim 5, characterized in that: The grouping rules for thread mapping obtained by offline processing based on the dequantized data layout and the layout required by the computing kernel specifically include: Extracting the dequantized data and initializing the computing thread; The computing threads are grouped according to the required dequantized data to obtain mini-thread bundles; A grouping rule for thread mapping is obtained based on the mini-warps and their corresponding dequantized data, wherein the grouping rule includes a shuffle offset.

7. The method for allocating operator fusion mechanisms for vector quantized large language models according to claim 6, characterized in that: The inter-thread register exchange is completed based on the dequantized data combined with the grouping rule, specifically including: The dequantized data in different threads are shuffled according to the grouping rule to complete the inter-thread register exchange, wherein: When shuffling data, a preset interface formula is used for shuffling, and the interface formula is as follows: Register ← shuffle xor (register, shuffle offset); Specifically, during the exchange, the original register value of the thread is exchanged with the register value of the corresponding shuffle offset.

8. An operator fusion mechanism allocation system for a vector quantized large language model, characterized in that: include: An acquisition module, used for acquiring layout data, wherein the layout data includes a dequantized data layout and a layout required for a computing kernel; A comparison module, configured to calculate the number of data shuffles based on the layout data, and compare the number of data shuffles with a preset threshold value to dynamically respond to a decision mechanism; A module is used to adopt a first decision mechanism when the number of data shuffles is less than the preset threshold, and the first decision mechanism at least includes a register-level operator fusion mechanism; and when the number of data shuffles is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least includes a shared memory-level operator fusion mechanism.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the operator fusion mechanism allocation method of the vector quantization large language model described in any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the operator fusion mechanism allocation method for the vector quantized large language model as described in any one of claims 1 to 7.

Images