Vector quantization operator fusion mechanism assignment method and system for large language models

Abstract

The application provides an operator fusion mechanism allocation method and system for vector quantization of a large language model, the method comprising the following steps: obtaining layout data, wherein the layout data comprises dequantization data layout and layout required by a calculation kernel; calculating a data shuffle frequency based on the layout data, and comparing the data shuffle frequency with a preset threshold to dynamically respond to a decision mechanism; when the data shuffle frequency is less than the preset threshold, a first decision mechanism is adopted, and the first decision mechanism at least comprises a register-level operator fusion mechanism; when the data shuffle frequency is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least comprises a shared memory-level operator fusion mechanism. The operator fusion mechanism allocation method and system for vector quantization of a large language model solve the bandwidth bottleneck and delay problem caused by frequent transmission of data between shared memory and registers, thereby realizing more efficient dequantization and calculation fusion.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a vector quantization large language model operator fusion mechanism allocation method and system. BACKGROUND

[0002] Currently, dequantization and subsequent operator fusion are mainly performed by using the shared memory of a GPU, data exchange and storage are performed within a thread block, frequent access to global memory is avoided, and data transmission efficiency is improved. As an intermediate cache, the shared memory allows threads to quickly exchange data without frequent access to global memory. This storage method takes advantage of the low latency of shared memory, greatly reduces the occupancy of memory bandwidth, and after storing the sub-vectors in the shared memory, each thread will extract the required specific data from the shared memory according to the requirements of the computing task.

[0003] The existing shared memory level fusion scheme uses shared memory as an intermediary to adjust the data layout after dequantization to adapt to the computing requirements. Although this method solves the problem of layout mismatch, it also causes a new bottleneck: frequent data transmission between shared memory and registers. Each data transmission increases the communication volume, resulting in frequent and inefficient transmission between registers and shared memory, forming a significant performance bottleneck. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a vector quantization large language model operator fusion mechanism allocation method and system to solve the problem of frequent communication transmission and low efficiency in the above-mentioned shared memory level fusion scheme.

[0005] In a first aspect, the present application provides a vector quantization large language model operator fusion mechanism allocation method, which comprises the following steps:

[0006] Obtaining layout data, wherein the layout data comprises dequantization data layout and computing kernel required layout;

[0007] Based on the layout data, the number of data shuffling is calculated, and the number of data shuffling is compared with a preset threshold to dynamically respond to the decision mechanism, wherein,

[0008] When the number of data shuffling is less than the preset threshold, a first decision mechanism is adopted, and the first decision mechanism at least comprises a register level operator fusion mechanism;

[0009] When the number of data shuffling is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least comprises a shared memory level operator fusion mechanism.

[0010] In a possible implementation of the present application, the obtaining of the layout data specifically includes:

[0011] obtaining the dequantized data layout based on the size of the sub-vectors in vector quantization;

[0012] obtaining the required layout of the computing kernel based on the specification of the current computing operator.

[0013] In a possible implementation of the present application, the calculating of the data shuffle times based on the layout data specifically includes:

[0014] calculating the size of a thread bundle based on the layout data, wherein the size of the thread bundle is obtained based on the ratio of the dequantized data layout to the required layout of the computing kernel;

[0015] calculating the data shuffle times based on the size of the thread bundle, wherein the data shuffle times are obtained by subtracting one from the number of the thread bundle.

[0016] In a possible implementation of the present application, the dynamic response decision mechanism specifically includes:

[0017] extracting the data shuffle times N and the preset threshold N t , wherein the preset threshold N t = 5.

[0018] calculating the size of the data shuffle times N and the preset threshold N t , wherein

[0019] when N < 5, the register-level operator fusion mechanism is adopted for operator calculation;

[0020] when N ≥ 5, the shared memory-level operator fusion mechanism is adopted for operator calculation.

[0021] In a possible implementation of the present application, the operator calculation based on the register-level operator fusion mechanism specifically includes:

[0022] obtaining grouping rules of thread mapping based on the dequantized data layout and the required layout of the computing kernel through offline processing;

[0023] dequantizing the quantized data by using the grouping rules of thread mapping to obtain the dequantized data;

[0024] completing inter-thread register exchange based on the dequantized data and the grouping rules;

[0025] after the completion of the inter-thread register exchange, the calculation of the computing operator is performed.

[0026] In a possible implementation of the present application, the thread mapping grouping rules obtained by offline processing based on the dequantized data layout and the layout required by the computing kernel specifically include:

[0027] Extracting the dequantized data and initializing the computing thread;

[0028] The computing threads are grouped according to the required dequantized data to obtain mini-thread bundles;

[0029] A grouping rule for thread mapping is obtained based on the mini-warps and their corresponding dequantized data, wherein the grouping rule includes a shuffle offset.

[0030] In a possible implementation of the present application, completing inter-thread register exchange based on the dequantized data in combination with the grouping rule specifically includes:

[0031] The dequantized data in different threads are shuffled according to the grouping rule to complete the inter-thread register exchange, wherein:

[0032] When shuffling data, a preset interface formula is used for shuffling. The interface formula is as follows:

[0033] Register ← shuffle xor (register, shuffle offset);

[0034] Specifically, during the exchange, the original register value of the thread is exchanged with the register value of the corresponding shuffle offset.

[0035] In a second aspect, the present invention provides an operator fusion mechanism allocation system for a vector quantized large language model, the system comprising:

[0036] An acquisition module, configured to acquire layout data, wherein the layout data includes a dequantized data layout and a layout required by a calculation kernel;

[0037] a comparison module, configured to calculate a data shuffling count based on the layout data, and compare the data shuffling count with a preset threshold value to dynamically respond to a decision-making mechanism;

[0038] A module is adopted, which is used to adopt a first decision mechanism when the number of data shuffling times is less than the preset threshold, and the first decision mechanism at least includes a register-level operator fusion mechanism; and when the number of data shuffling times is greater than or equal to the preset threshold, adopt a second decision mechanism, and the second decision mechanism at least includes a shared memory-level operator fusion mechanism.

[0039] In a third aspect, the present invention provides an electronic device, comprising: a processor and a memory;

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program stored in the memory so that the electronic device performs the above-mentioned operator fusion mechanism allocation method for the vector quantization large language model.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned method for allocating an operator fusion mechanism for a vector quantized large language model.

[0043] As described above, the method and system for allocating an operator fusion mechanism for a vector quantized large language model according to the present invention have the following beneficial effects:

[0044] (1) Register-level fusion can directly perform shuffle operations between mini-warps, requiring only one data transfer, greatly reducing operation latency;

[0045] (2) Register-level fusion reduces the need for synchronization instructions through direct register exchange, further improving efficiency;

[0046] (3) The hierarchical adaptive operator fusion allocation can maximize the utilization of hardware resources and avoid the performance bottleneck caused by fixed fusion strategies, making it highly versatile when processing different vector quantization schemes and different types of operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Shown is a schematic diagram of a scene of an electronic device in an embodiment of the present invention;

[0048] Figure 2 Shown is a flow chart of an embodiment of a method for allocating an operator fusion mechanism for a vector quantized large language model according to the present invention;

[0049] Figure 3 A schematic diagram showing the steps of a dynamic response decision mechanism in an embodiment of the method for allocating an operator fusion mechanism for a vector quantized large language model of the present invention;

[0050] Figure 4 A schematic diagram showing steps corresponding to the fusion mechanism of the first decision mechanism and the second decision mechanism in an embodiment of the operator fusion mechanism allocation method for a vector quantized large language model of the present invention;

[0051] Figure 5 An example diagram showing data exchange within a warp based on data shuffling in an embodiment of an operator fusion mechanism allocation method for a vector quantized large language model of the present invention;

[0052] Figure 6 Shown is a schematic structural diagram of an operator fusion mechanism allocation system for a vector quantized large language model according to an embodiment of the present invention;

[0053] Figure 7 FIG. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0055] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0056] The following embodiments of the present invention provide a method for allocating an operator fusion mechanism for a vector quantized large language model, which can be applied to Figure 1 The electronic device shown. The electronic device described in the present invention may include a mobile phone 11 with wireless charging function, a tablet computer 12, a laptop computer 13, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present invention does not impose any restrictions on the specific type of electronic device.

[0057] For example, the electronic device may be a station (ST) in a WLAN with a wireless charging function, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a personal digital assistant (PDA) device, a handheld device with a wireless charging function, a computing device or other processing device, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next-generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future-evolved public land mobile network (PLMN), or a mobile terminal in a future-evolved non-terrestrial network (NTN).

[0058] With the rapid development of artificial intelligence, large language models (such as GPT and BERT) have demonstrated tremendous potential in natural language processing, machine translation, and conversational systems. However, as models scale, reaching billions or even hundreds of billions of parameters, the demand for computing resources and memory increases dramatically, placing significant pressure on hardware. This is especially true during inference, where frequent data loading and processing increases computational latency. Therefore, reducing memory usage and accelerating inference have become key challenges in deploying large models.

[0059] In order to address the resource demand challenges brought by large models, quantization technology has become one of the important means to optimize computing and storage resources. Quantization technology significantly reduces the storage requirements of the model by converting high-precision floating-point numbers (such as FP16 or FP32) in the model into low-precision representations (such as INT8, INT4). It is worth noting that although this element-by-element quantization method is relatively simple, it can only achieve a very limited compression rate because it does not consider the correlation information between elements. Among them, when the quantization bit width is further reduced, the model accuracy is prone to a significant decline. In order to overcome the shortcomings of element-by-element quantization, vector quantization technology came into being. Vector quantization can effectively solve the shortcomings of element-by-element quantization in compression rate, quantization error and structural information capture, and has shown significant potential in accelerating the inference of large language models.

[0060] Unlike element-by-element quantization, vector quantization maps a set of high-dimensional vectors to a finite set of discrete codewords, effectively compressing the data representation through a codebook. Specifically, vector quantization technology first generates a codebook containing representative vectors using a clustering algorithm. During quantization, each vector is replaced with its closest codebook index. During dequantization, the corresponding codeword is searched from the codebook using the index, thereby recovering an approximate image of the original data. The following describes the technical solutions in the embodiments of the present invention in detail, in conjunction with the accompanying drawings.

[0061] Vector quantization is mainly used in large models to compress weights and KV cache (Key-Value Cache). Weights are the part of the model parameters that takes up a large amount of storage space, while KV cache is used to store intermediate calculation results during the model inference process. These are all heavily dependent on memory. However, although vector quantization performs well in compression, translating this into an increase in inference speed is a challenge. In the inference process, the compressed data must first be dequantized before participating in the calculation, that is, the index is restored to the original vector by searching the codebook. This process often needs to be performed at each stage of inference because the quantized data cannot directly participate in matrix multiplication or other operations. Therefore, optimizing the fusion kernel and more efficiently integrating the dequantization and calculation processes can not only reduce memory access latency but also significantly improve data processing speed.

[0062] Currently, the fusion of devector quantization and subsequent operators primarily utilizes the GPU's shared memory to exchange and store data within thread blocks. Specifically, the bandwidth of shared memory is limited, making it unable to support the high-frequency data exchange required for large-scale parallel computing. Frequent data transfer consumes valuable bandwidth resources, further reducing computational efficiency. Furthermore, shared memory access speed is much slower than registers, introducing latency each time data is transferred from shared memory to registers for computation. These latency accumulates in large-scale parallel computing, leading to underutilization of computing resources and reduced overall system performance. Furthermore, the capacity of shared memory is limited. Especially when processing high-dimensional vectors and large amounts of data, the storage and transfer of dequantization results is limited not only by the shared memory's bandwidth but also by its storage capacity. Shared memory cannot accommodate all dequantized data at once, meaning data must be loaded and transferred in batches, further increasing system latency and complexity.

[0063] To this end, the present invention introduces a hierarchical operator fusion scheme, which expands register-level fusion on the basis of shared memory-level fusion. It can dynamically select the optimal scheme based on the overhead of the two schemes, solving the bandwidth bottleneck and delay problems caused by frequent data transmission between shared memory and registers, thereby achieving more efficient dequantization and computational fusion.

[0064] Specifically, see Figure 2 In one embodiment of the invention, the method for allocating an operator fusion mechanism for a vector quantized large language model of the present invention includes the following steps:

[0065] Step S202, obtaining layout data, wherein the layout data includes a dequantized data layout and a layout required by a calculation kernel;

[0066] Step S204, calculating the number of data shuffling times based on the layout data, and comparing the number of data shuffling times with a preset threshold to dynamically respond to a decision-making mechanism;

[0067] Step S206: When the number of data shuffling times is less than the preset threshold, a first decision mechanism is adopted, wherein the first decision mechanism at least includes a register-level operator fusion mechanism;

[0068] Step S208: When the number of data shuffling times is greater than or equal to the preset threshold, a second decision mechanism is adopted, where the second decision mechanism at least includes a shared memory level operator fusion mechanism.

[0069] It should be noted that the present invention provides an expanded register-level fusion solution. Through register-level data exchange, it effectively bypasses the frequent transmission delays caused by shared memory computing in certain application scenarios, realizes the data layout conversion from dequantization to operator calculation, improves the fusion efficiency, and realizes an adaptive dynamic decision-making mechanism. It automatically selects shared memory-level fusion or register-level fusion according to the dequantization data layout and the layout required by the computing kernel.

[0070] Specifically, in this embodiment, layout data is first obtained. Accordingly, obtaining the layout data specifically includes: obtaining the dequantized data layout based on the size of the sub-vector in the vector quantization, and obtaining the required layout of the computing kernel based on the specifications of the current computing operator, wherein the size of the sub-vector and the specifications of the computing operator are known given quantities.

[0071] Furthermore, after obtaining the layout data, the number of data shufflings can be obtained by calculation based on the layout data. The subsequent specification will provide a detailed explanation. After obtaining the number of data shufflings, it can be compared with a preset threshold to perform a dynamic response decision mechanism. The greater the difference between the dequantized data layout and the layout required by the computing kernel, the greater the demand for data shuffling. When the number of required data exchanges exceeds a certain threshold, the overhead will be higher than the data exchange at the shared memory level. Therefore, it is necessary to calculate the number of data shufflings based on the layout data to adaptively adjust the level of operator fusion.

[0072] Specifically, experiments have found that the access latency of shared memory is almost five times that of the combination of register access and shuffle operations. Therefore, for quantized tensors that require less than "5" shuffle operations, a register-level fusion method can be used. For tensors that require more shuffle operations, the traditional shared memory-level fusion scheme is maintained. Through adaptive decision-making, the optimal fusion method is effectively selected according to the characteristics of different quantized tensors, which can optimize system efficiency while ensuring performance.

[0073] Furthermore, if Figure 3 FIG. 1 shows a schematic diagram of a dynamic response decision-making mechanism, which specifically includes the following steps:

[0074] Step 1: Extract the data shuffling times N and the preset threshold N t , wherein the preset threshold N t =5;

[0075] Step 2: Calculate the data shuffling times N and the preset threshold N t The size of which enters the judgment branch, specifically,

[0076] When N<5, the first decision mechanism is adopted, that is, the register-level operator fusion mechanism is adopted to perform operator calculation;

[0077] When N≥5, the second decision mechanism is sampled, that is, the shared memory level operator fusion mechanism is used to perform operator calculation.

[0078] Specifically, if Figure 4 As shown, it is a schematic diagram of the steps corresponding to the fusion mechanism of the first decision mechanism and the second decision mechanism, wherein: Figure 4 The right branch corresponds to the steps of the register-level operator fusion mechanism shown in the first decision mechanism. Figure 4 The left branch corresponds to the steps of the shared memory level operator fusion mechanism indicated by the second decision mechanism.

[0079] Furthermore, in one embodiment of the invention, the calculating the number of data shuffling times based on the layout data specifically includes:

[0080] Calculating a size of a thread warp based on the layout data, wherein the size of the thread warp is obtained based on a ratio of the dequantized data layout to a layout required by the computing kernel;

[0081] The number of data shuffles is obtained by calculating based on the size of the thread warp, wherein the number of data shuffles is obtained by subtracting one from the number of the thread warp.

[0082] It should be noted that, in this embodiment, Figure 5For example, an example diagram of data exchange within a thread bundle based on data shuffling is shown, wherein the quantized data is a "16×2" matrix, which needs to be dequantized to obtain the original "16×16" layout. Therefore, the dequantized data layout is "8", indicating that the register of a thread will hold "8" consecutive dequantized data, and the required layout of the corresponding computing kernel is "2", that is, each thread only needs to hold "2" consecutive elements to complete the calculation. Therefore, based on the ratio of the dequantized data layout to the required layout of the computing kernel, the thread bundle is obtained, and the size of the thread bundle is "4", and the number of shuffles is the number of the thread bundle minus one. Therefore, for Figure 5 For the example scenario, the number of shuffling times corresponds to "3".

[0083] Furthermore, in one embodiment of the invention, operator calculation is performed based on a register-level operator fusion mechanism, specifically including:

[0084] The thread mapping grouping rules are obtained by offline processing based on the dequantized data layout and the layout required by the computing kernel;

[0085] Dequantizing the quantized data using the grouping rule of the thread mapping to obtain the dequantized data;

[0086] completing inter-thread register exchange based on the dequantized data in combination with the grouping rule;

[0087] After completing the register exchange between threads, the calculation of the calculation operator is performed.

[0088] It should be noted that, in this embodiment,

[0089] The thread mapping grouping rule is obtained by performing offline processing based on the dequantized data layout and the required layout of the computing kernel, specifically comprising: extracting the dequantized data, initializing the computing threads, and grouping the computing threads according to the required dequantized data to obtain mini-thread bundles, wherein the mini-thread bundles correspond to the dequantized data required for processing after grouping, and then obtaining the thread mapping grouping rule based on the mini-thread bundles and their corresponding dequantized data, wherein the grouping rule includes a shuffle offset, wherein, for example, Figure 5 As shown, the thread program numbers in the mini-thread bundle include "sequence number 0, sequence number 1, sequence number 2 and sequence number 3", the color block corresponds to the dequantized data to be processed, and then the grouping rule is obtained based on the mini-thread bundle and the corresponding dequantized data, which specifically corresponds to the offset rule between different thread program numbers. Specifically, the grouping rule includes a shuffle offset.

[0090] Furthermore, completing inter-thread register exchange based on the dequantized data in combination with the grouping rule specifically includes:

[0091] The dequantized data in different threads are shuffled according to the grouping rule to complete the inter-thread register exchange. When shuffling the data, a preset interface formula is used for shuffling. The interface formula is as follows:

[0092] Register←shuffle xor (register, shuffle offset);

[0093] Specifically, during the exchange, the original register value of the thread is exchanged with the register value of the corresponding shuffle offset.

[0094] Specifically, shuffle means shuffle, and xor means exclusive OR operation. The exchange operation is processed offline in the register, and data exchange can be realized in the register device without transferring through shared memory, thereby reducing system latency. Specifically, Figure 5 As shown, for a warp of "4", "3" shuffle operations need to be performed. Each shuffle is performed based on the corresponding shuffle offset. The process is as follows:

[0095]

[0096] Among them, Tid X.[Y] represents the Yth element of the register array in the thread with thread program number X. For example, Tid0.[1] represents the element with register array index "1" in the thread with "serial number 0". After "3" times of shuffling, it is ensured that the data required by the thread with each serial number during calculation can be dequantized in the thread bundle in which it is located without relying on the data of other thread bundles. In this way, the entire data processing process is optimized, the redundant data exchange between threads is reduced, and the overall computing efficiency is improved.

[0097] The example illustrated in the above embodiment corresponds to the first decision mechanism, that is, it corresponds to the register-level operator fusion mechanism, and when N≥5, the second decision mechanism is sampled, that is, the shared memory-level operator fusion mechanism is used to perform operator calculation. When shared memory is used for data exchange and storage, for the scenario where the dequantized data layout is "8", each sub-vector is represented by a quantization index. During the dequantization process, the "32" parallel threads in the thread block are used to read the "8"-dimensional sub-vector from the codebook through the quantization index, and then the dequantized sub-vectors are exchanged between different threads. Since the shared memory-level operator fusion mechanism is a technical means that can be directly applied by technicians in this field, it will not be elaborated here.

[0098] An embodiment of the present application also provides an operator fusion mechanism allocation system for a vector quantized large language model. The operator fusion mechanism allocation system for a vector quantized large language model can implement the operator fusion mechanism allocation method for a vector quantized large language model described in the present application. However, the implementation device of the operator fusion mechanism allocation method for a vector quantized large language model described in the present application includes but is not limited to the structure of the operator fusion mechanism allocation system for a vector quantized large language model listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0099] See also Figure 6 In one embodiment, this embodiment provides an operator fusion mechanism allocation system 60 for a vector quantized large language model, the system comprising:

[0100] An acquisition module 61 is configured to acquire layout data, wherein the layout data includes a dequantized data layout and a layout required by a calculation kernel;

[0101] a comparison module 62 for calculating a data shuffling count based on the layout data, and comparing the data shuffling count with a preset threshold value to dynamically respond to a decision-making mechanism;

[0102] Module 63 is used to adopt a first decision mechanism when the number of data shuffling times is less than the preset threshold, and the first decision mechanism at least includes a register-level operator fusion mechanism; and when the number of data shuffling times is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least includes a shared memory-level operator fusion mechanism.

[0103] Since the specific implementation of this embodiment corresponds to the aforementioned method embodiment, the same details will not be repeated here, and those skilled in the art should also understand that Figure 6 The division of the various modules in the embodiment is merely a division of logical functions. In actual implementation, all or part of the modules can be integrated into one or more physical entities, and all of these modules can be implemented in the form of software called by processing elements, or all of them can be implemented in the form of hardware. Some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware.

[0104] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices or methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.

[0105] Modules / units described as separate components may or may not be physically separate, and components displayed as modules / units may or may not be physical modules, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules / units may be selected based on actual needs to achieve the objectives of the embodiments of the present invention. For example, the functional modules / units in various embodiments of the present invention may be integrated into a single processing module, each module / unit may exist physically separately, or two or more modules / units may be integrated into a single module / unit.

[0106] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0107] The embodiment of the present invention also provides a computer-readable storage medium. A person skilled in the art will understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state drive, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid-state drive (SSD)), etc.

[0108] An embodiment of the present invention further provides an electronic device comprising a processor and a memory.

[0109] The memory is used to store computer programs.

[0110] The memory includes various media that can store program codes, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0111] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned operator fusion mechanism allocation method for the vector quantization large language model.

[0112] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0113] like Figure 7As shown, the electronic device of the present invention is in the form of a general-purpose computing device. Components of the electronic device may include, but are not limited to: one or more processors or processing units 71, a memory 72, and a bus 73 connecting different system components (including the memory 72 and the processing unit 71).

[0114] Bus 73 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0115] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0116] The memory 72 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 721 and / or cache memory 722. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 723 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 73 via one or more data medium interfaces. Memory 72 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0117] A program / utility 724 having a set (at least one) of program modules 7241 may be stored, for example, in memory 72. Such program modules 7241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 7241 generally implement the functions and / or methods of the embodiments described herein.

[0118] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 74. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 75. Figure 7 As shown, the network adapter 75 communicates with other modules of the electronic device via the bus 73. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A method for allocating an operator fusion mechanism for a vector quantized large language model, characterized in that: include: Obtaining layout data, wherein the layout data includes a dequantized data layout and a layout required by a computing kernel, specifically comprising: obtaining the dequantized data layout based on the size of a subvector in vector quantization; obtaining the layout required by the computing kernel based on a specification of a current computing operator; The number of data shuffles is calculated based on the layout data, wherein the number of data shuffles is calculated based on the layout data, specifically comprising: calculating a size of a thread warp based on the layout data, wherein the size of the thread warp is obtained based on a ratio of the dequantized data layout to the layout required by the computing kernel; calculating the number of data shuffles based on the size of the thread warp, wherein the number of data shuffles is obtained by subtracting one from the number of the thread warps, and comparing the number of data shuffles with a preset threshold to dynamically respond to a decision mechanism, wherein when the number of data shuffles is less than the preset threshold, a first decision mechanism is adopted, wherein the first decision mechanism at least includes a register-level operator fusion mechanism; When the number of data shuffling times is greater than or equal to the preset threshold, a second decision mechanism is adopted, and the second decision mechanism at least includes a shared memory level operator fusion mechanism.

2. The method for allocating an operator fusion mechanism for a vector quantized large language model according to claim 1, wherein: The dynamic response decision-making mechanism specifically includes: Extract the data shuffling times N and the preset threshold N t , wherein the preset threshold N t =5; Calculate the data shuffling times N and the preset threshold N t The size of When N<5, the register-level operator fusion mechanism is used to perform operator calculations; When N≥5, the shared memory level operator fusion mechanism is used to perform operator calculation.

3. The method for allocating an operator fusion mechanism for a vector quantized large language model according to claim 2, wherein: Operator calculations are performed based on the register-level operator fusion mechanism, specifically including: The thread mapping grouping rules are obtained by offline processing based on the dequantized data layout and the layout required by the computing kernel; Dequantizing the quantized data using the grouping rule of thread mapping to obtain the dequantized data; completing inter-thread register exchange based on the dequantized data in combination with the grouping rule; After completing the register exchange between threads, the calculation of the calculation operator is performed.

4. The method for allocating an operator fusion mechanism for a vector quantized large language model according to claim 3, wherein: The thread mapping grouping rules obtained by offline processing based on the dequantized data layout and the layout required by the computing kernel specifically include: Extracting the dequantized data and initializing the computing thread; The computing threads are grouped according to the required dequantized data to obtain mini-thread bundles; A grouping rule for thread mapping is obtained based on the mini-warps and their corresponding dequantized data, wherein the grouping rule includes a shuffle offset.

5. The method for allocating an operator fusion mechanism for a vector quantized large language model according to claim 4, wherein: The inter-thread register exchange is completed based on the dequantized data and the grouping rule, specifically including: The dequantized data in different threads are shuffled according to the grouping rule to complete the inter-thread register exchange, wherein: When shuffling data, a preset interface formula is used for shuffling. The interface formula is as follows: Register ← shuffle xor (register, shuffle offset); Specifically, during the exchange, the original register value of the thread is exchanged with the register value of the corresponding shuffle offset.

6. A system for distributing operator fusion mechanisms for vector quantized large language models, characterized in that: include: an acquisition module, configured to acquire layout data, wherein the layout data includes a dequantized data layout and a layout required by a computing kernel, specifically comprising: obtaining the dequantized data layout based on the size of a subvector in vector quantization; and obtaining the layout required by the computing kernel based on the specification of a current computing operator; a comparison module, configured to calculate a number of data shuffles based on the layout data, wherein the calculating the number of data shuffles based on the layout data specifically comprises: calculating a size of a thread bundle based on the layout data, wherein the size of the thread bundle is obtained based on a ratio of the dequantized data layout to a layout required by the computing kernel; calculating the number of data shuffles based on the size of the thread bundle, wherein the number of data shuffles is obtained by subtracting one from the number of thread bundles; and comparing the number of data shuffles with a preset threshold to dynamically respond to a decision mechanism; A module is adopted, which is used to adopt a first decision mechanism when the number of data shuffling times is less than the preset threshold, and the first decision mechanism at least includes a register-level operator fusion mechanism; and when the number of data shuffling times is greater than or equal to the preset threshold, adopt a second decision mechanism, and the second decision mechanism at least includes a shared memory-level operator fusion mechanism.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the operator fusion mechanism allocation method of the vector quantized large language model described in any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: The electronic device includes: a processor and a memory; wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the operator fusion mechanism allocation method for the vector quantized large language model as described in any one of claims 1 to 5.

Images