Data processing method based on large language model, and large language model and electronic device

Abstract

Disclosed in the present application are a data processing method based on a large language model, and a large language model, an electronic device, a computer-readable storage medium and a computer program product. The method is applied to a user terminal, wherein a large language model is deployed on the user terminal, and weight parameters of linear calculation layers of the large language model are pre-quantized into format data of an integer data type. The method comprises: acquiring input data; performing vector conversion on the input data by means of an embedding layer of a large language model, so as to obtain a floating-point query vector of a floating-point data type corresponding to the input data; converting the floating-point query vector into an integer query vector of an integer data type; and performing an operation by means of weight parameters of linear calculation layers and the integer query vector, so as to obtain a query result corresponding to the input data. By means of the solution provided in the present application, a large language model can be smoothly run on a user terminal, such that the user terminal can provide services for users without needing network connectivity, and can better ensure the privacy of the users.

Description

Method for processing data based on large language model, large language model and electronic device

[0001] Cross-references

[0002] This application refers to Chinese Patent Application No. 2024103986001 filed on April 2, 2024, entitled “Method for processing data based on a large language model, large language model and electronic device”, which is incorporated into this application in its entirety by reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a method for processing data based on a large language model, a large language model, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0004] With the rapid development of computer technology, large language models, with their superior data processing capabilities, can be used in a wide range of scenarios, such as generating required images, querying professional knowledge, engaging in conversations with large language models, and generating qualified articles. However, due to the large number of parameters and computational complexity of large language models, which require significant computing resources, large language models are typically deployed on cloud servers.

[0005] However, when the large language model is deployed on the cloud server, the user terminal needs to be connected to the network before it can use the large language model. In addition, since the user needs to upload text information during use, such as user questions and developer code, to the cloud server, there is a certain risk of leakage of the user's privacy information, which makes it inconvenient for the user to use the large language model on the user terminal. Summary of the Invention

[0006] The present application provides a method for processing data based on a large language model, a large language model, an electronic device, a computer-readable storage medium, and a computer program product, which can smoothly run the large language model on a user terminal, thereby enabling the user terminal to provide services to the user without being connected to the Internet and better protecting the user's privacy.

[0007] In a first aspect, the present application provides a method for processing data based on a large language model, which is applied to a user terminal, wherein a large language model is deployed on the user terminal, and the weight parameters of each linear calculation layer of the large language model are pre-quantized into integer data type format; the method comprises:

[0008] Get input data;

[0009] Performing vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data;

[0010] Converting the floating-point query vector to an integer query vector of an integer data type;

[0011] The query result corresponding to the input data is obtained by performing an operation on the weight parameter of the linear calculation layer and the integer query vector.

[0012] In a second aspect, the present application provides a large language model, including:

[0013] A text conversion tokenizer layer, used to convert input data into a text data format supported by the large language model;

[0014] Embedding layer, used to convert text data into vectors;

[0015] The Transformer layer includes various linear calculation Linear layers, the weight parameters of the Linear layers are quantized into integer data type format data, and the Linear layers are used to perform linear transformation on the input vector data to obtain the model operation results.

[0016] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor; when the processor executes the computer program instructions, the method as described in any one of the first aspects is implemented.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any of the methods described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method as described in any one of the first aspects.

[0019] Compared with the prior art, this application has the following advantages:

[0020] The method for processing data based on a large language model provided in an embodiment of the present application is applied to a user terminal. A large language model is deployed on the user terminal, and the weight parameters of each linear calculation layer of the large language model are pre-quantized into data in the format of an integer data type. Since the weight parameters of each linear calculation layer are quantized into integer data types, compared with weight parameters of floating-point data types, the weight parameters of integer data types can significantly reduce the amount of memory access when performing matrix operations. Since the memory of the user terminal is usually relatively small, using weight parameters of integer data types to perform matrix operations on input data can significantly improve the model's calculation and memory access efficiency, thereby improving the calculation efficiency of the large language model and the requirements for device resources, allowing the user terminal to run the large language model more smoothly. The weight parameters of integer data types can also effectively replace the information of weight parameters of floating-point data types, ensuring calculation accuracy. When processing data through a large language model, the solution provided in the present application first performs vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data, and then converts the floating-point query vector into an integer query vector of an integer data type. The integer query vector is operated on by the weight parameters of the linear calculation layer to obtain a query result corresponding to the input data. When the integer query vector of the integer data type of the input data performs operations such as matrix multiplication in the linear calculation layer, the memory access amount can be further reduced, thereby improving the computing efficiency of the user terminal.

[0021] It can be seen that the solution provided by the present application deploys a large language model on a user terminal for operation. By quantizing the weight parameters of the large language model and the input data entered by the user into data of integer data types, the amount of memory access and the amount of calculation required by the large language model when running on the user terminal can be greatly reduced, making it more suitable for situations where the user terminal has small memory and limited computing resources, so that the user terminal can run the large language model in the streaming field. In addition, since the distribution difference of the weight parameters of the linear calculation layer of the large language model is usually very small, the distribution similarity of each weight parameter after quantization into an integer type is very high. Therefore, the quantization of the weight parameter has little effect on the calculation accuracy of the large language model, and since the integer data type usually has a sufficient range to represent the input data of the large language model, the query vector corresponding to the input data can be quantized very completely through the integer data type, thereby better ensuring that the information of the input data is not lost. Therefore, the weight parameter quantization and query vector quantization performed in this application can well ensure the computational accuracy of the large language model running on the user terminal. In this way, since the large language model is deployed on the user terminal, the user terminal can reply to the user's query content without being connected to the Internet, which better ensures the privacy of the user's query content and is more convenient for users to use. In addition, since there is no need to equip a dedicated cloud server to deploy the large language model, the operating cost of the large language model can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] FIG1 is a schematic diagram of an application scenario of a solution for processing data based on a large language model provided by the present application;

[0024] FIG2 is a flow chart of an example of a method for processing data based on a large language model provided in an embodiment of the present application;

[0025] FIG3 is an abstract structural diagram of a large language model in an embodiment of the present application;

[0026] FIG4 is an example diagram of various functional modules when exporting a large language model in an embodiment of the present application;

[0027] FIG5 is an example diagram of model calculation using the W4A8 quantization strategy in an embodiment of the present application;

[0028] FIG6 is a block diagram of a device for processing data based on a large language model according to an embodiment of the present application;

[0029] FIG7 is a structural block diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions of this application, the following clearly and completely describes this application in conjunction with the drawings in the embodiments of this application. However, this application can be implemented in many other ways different from the following description. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of this application.

[0031] It should be noted that the terms "first", "source domain", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] In order to facilitate understanding of the various embodiments of the present application, the application background of the embodiments is described.

[0033] With the rapid development of computer technology, large language models, due to their outstanding data processing capabilities, can demonstrate their capabilities in many scenarios, such as generating required images through large language models, querying professional knowledge through large language models, communicating with large language models, and generating articles that meet requirements through large language models.

[0034] Large language models are natural language processing models with a large number of parameters and capabilities. When processing text and other related tasks, these models can leverage their large-scale parameters and data to generate more accurate and coherent text output. For example, large language models based on neural networks (such as the GPT series of models and the BERT model) can automatically understand, generate, and infer text by training on massive amounts of language data. They are suitable for a variety of tasks such as language translation, text summarization, and dialogue systems.

[0035] Since large language models have a large number of parameters and computational complexity and require relatively large computing resources, they are usually deployed on cloud servers.

[0036] However, when large language models are deployed on a cloud server, user terminals must be connected to the internet to use them. Furthermore, since users must upload text information, such as user questions and developer code, to the cloud server, this poses a risk of privacy leakage, making it inconvenient for users to use large language models on their terminals. Furthermore, a dedicated cloud server is required to deploy large language models, increasing their operating costs.

[0037] To address the above issues, embodiments of the present application provide a method, apparatus, electronic device, and computer-readable storage medium for processing data based on a large language model. These methods aim to enable the smooth operation of large language models on user terminals, enabling them to provide services without requiring an internet connection, better protecting user privacy, and reducing the cost of deploying and operating large language models.

[0038] The method for processing data based on a large language model provided in this application can be applied to deploying and running a large language model on a user terminal. The user terminal can be, for example, a mobile terminal such as a mobile phone, a smart watch, a smart VR device, a smart car device, a laptop computer, etc., or a desktop computer or other non-mobile user terminal.

[0039] In order to facilitate understanding of the method embodiment of the present application, its application scenario is introduced. Please refer to Figure 1, which is a schematic diagram of the application scenario of the solution provided by the embodiment of the present application. It can be applied to a user terminal 101, which can be a mobile phone, tablet computer (pad), vehicle-mounted device, wearable device, smart TV, virtual reality VR device, laptop computer, desktop computer and other devices capable of running a large language model. A large language model is pre-deployed on the user terminal 101, and the weight parameters of each linear calculation layer of the large language model are pre-quantized into format data of an integer data type.

[0040] In this embodiment, after the user enters the query content on the user terminal 101, the user terminal 101 can determine the input data entered by the user, and perform inference processing on the input data through the large language model on the user terminal 101 to obtain the query result corresponding to the input data.

[0041] Example 1

[0042] The first embodiment of the present application provides a method for processing data based on a large language model, which is applied to a user terminal. Specifically, the execution subject of the method can be an electronic device used by the user and deployed with a large language model. The electronic device can be a desktop computer, a laptop computer, a smart mobile terminal, a client device, etc., or other electronic devices with data processing capabilities and capable of a large language model. A large language model is deployed on the above-mentioned user terminal, and the weight parameters of each linear calculation layer of the large language model are pre-quantized into format data of an integer data type.

[0043] It can be understood that a large language model (LLM) usually includes a text conversion tokenizer layer, an embedding layer, a Transformer layer, etc., among which the Transformer layer includes a linear calculation Linear layer, which is the part of the Transformer layer that contains weight parameters.

[0044] The tokenizer layer is used to convert the input text (i.e., the input data in this application) into a format that the model can understand and process. The main function of the tokenizer is to split the original text into words, subwords, or characters and map them to corresponding identifiers (tokens) in the model's vocabulary.

[0045] The embedding layer is used to convert text data into vectors to represent the concept of the text through vectors.

[0046] The Linear layer is used to perform linear transformations on input data. Specifically, it performs inference operations on the input data through operations such as matrix multiplication and bias addition to obtain model operation results that meet the requirements. The Linear layer is one of the basic operation operators of the large language model and is used to form the basic unit of the large language model. The main structure of the LLM is the decoder layer of the Transformer structure. The main weight parameters in the decoder layer are basically concentrated in the operators of the Linear layer.

[0047] Since the weight parameters of the linear layer are usually represented by floating-point data types, for example, by the 32-bit floating-point data type fp32, the memory access amount of the calculation process is very large, and the number of linear layers in a large language model is usually very large, usually including dozens of linear layers. Representing each linear layer by floating-point data types will require a large amount of computing resources and a large amount of memory access, which is not conducive to the flow field operation of the user terminal. Therefore, the weight parameters of each linear calculation layer can be pre-quantized into integer data type format data, thereby reducing the memory access and computational complexity during the linear operation process. Specifically, the weight parameters of the linear calculation layer can be pre-quantized into 4-bit binary integer int4 format data or 8-bit binary integer int8 format data. Since the int4 format data has a smaller computational complexity and smaller memory access during calculation, the weight parameters of the linear calculation layer can be pre-quantized into int4 format data.

[0048] The above-mentioned large language model may further include an attention layer, an activation function layer, etc. Those skilled in the art may set the various data processing layers included in the large language model according to specific scenarios.

[0049] In an embodiment of the present application, the tokenizer layer is used to convert text (string) into an integer (int), and the weight parameters of the embedding layer can be represented by a floating-point data type to improve the text conversion accuracy and vector conversion accuracy, thereby improving the reasoning accuracy of the model.

[0050] It is understandable that in a large language model, each layer has its own weight parameter, which can be a weight matrix. The weight matrix describes the linear mapping relationship between input and output, and the weight parameter determines the propagation and conversion process of the input data in the model network.

[0051] As shown in Figure 3, the abstract structure of a large language model typically includes an embedding layer, various data transformation layers (block-1, block-2, etc.), and a language model layer (LM) for evaluating text probability. The embedding layer is used to perform text vectorization conversion, and each block is used to perform operations based on the output data of the previous layer, and the output results are obtained through the LM layer.

[0052] As shown in FIG2 , the method for processing data based on a large language model provided in the first embodiment of the present application includes the following steps S110 to S140 .

[0053] Step S110: Obtain input data.

[0054] The above-mentioned input data is data entered by the user on the user terminal. For example, the input data may be "Please synthesize a picture", "Please explain the meaning of A", "Please generate a code to implement function a", "Please explain the execution logic of code b", "Please translate the sentence c", etc. The input data is the question data entered by the user corresponding to the user query content. The user terminal can process and infer the input data through the various data processing layers of the large language model to output an output result that matches the input data.

[0055] Step S120: performing vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data.

[0056] To improve model inference accuracy, the embedding layer of a large language model typically converts input data into floating-point query vectors of the floating-point data type. For example, the input data is converted into floating-point query vectors of the fp32 or fp16 floating-point type. The fp16 floating-point type has lower computational complexity and higher computational efficiency than the fp32 floating-point type. Therefore, it is recommended to convert the input data into floating-point query vectors of the fp16 floating-point type.

[0057] Optionally, step S120 may input the input data into the above embedding layer to obtain a floating-point query vector of a floating-point data type corresponding to the input data.

[0058] Step S130: Convert the floating-point query vector into an integer query vector of integer data type.

[0059] Specifically, the floating-point query vector can be converted into an integer query vector in the 8-bit binary integer int8 format or an integer query vector in the int4 format. Since the amount of data corresponding to the input data is usually relatively small, converting the floating-point query vector into an integer query vector of the int8 type can not only improve the computational efficiency, but also better ensure the accuracy of subsequent inference operations.

[0060] Step S140: Calculate the weight parameters of the linear calculation layer and the integer query vector to obtain a query result corresponding to the input data.

[0061] In an embodiment of the present application, when the floating-point query vector is converted into an integer query vector of type int8, and the weight parameter of the linear calculation layer is quantized into format data of type int4, the quantization method in this case can be referred to as adopting the W4A8 quantization strategy, where W represents weight, which refers to the weight parameter of the linear calculation layer of the large language model, 4 represents 4 bits, indicating that the weight parameter of the linear calculation layer is quantized to 4 bits for storage, that is, the weight parameter is quantized to the int4 data type, A represents activate, which refers to the activation value, that is, the input data at runtime (that is, the above-mentioned floating-point query vector), and 8 represents the use of 8 bits, that is, the activation value will be dynamically quantized from a floating-point number to 8 bits at runtime, that is, converted to an integer query vector of the int8 data type.

[0062] Specifically, the query result corresponding to the input data can be obtained by performing matrix multiplication on the weight matrix (ie, weight parameter) of the linear calculation layer and the integer query vector.

[0063] In a specific embodiment, step S140 can be implemented according to the following steps S141 to S144.

[0064] Step S141: converting the weight parameters of the linear calculation layer into weight parameters of a target integer type, where the target integer type is the same as the integer type corresponding to the integer query vector.

[0065] For example, when the integer query vector corresponds to the int8 type, the target integer type is also int8. Step S141 converts the weight matrix of the linear calculation layer into a data type that is the same as the integer query vector type corresponding to the input data, so as to facilitate subsequent matrix operations.

[0066] Step S142: performing matrix multiplication on the integer query vector and the weight parameter of the target integer type to obtain a matrix operation result.

[0067] Step S143: Dequantize the above matrix operation result to obtain an output result of floating-point data type.

[0068] Optionally, the matrix operation result may be dequantized through the following steps S143a to S143b.

[0069] Step S143a: Obtain the scaling factor and zero point when quantizing the weight parameters of the above-mentioned linear calculation layer, wherein the scaling factor is the scaling factor for mapping the floating-point number to the integer range, and the zero point is the offset for mapping the zero value of the floating-point number to the zero value of the integer.

[0070] Step S143b: Dequantize the matrix operation result according to the scaling factor and the zero point to obtain an output result of a floating-point data type.

[0071] Since the weight parameters of other processing layers in the large language model are usually floating-point weight parameters, the dequantization operation can ensure that other inference layers of the large language model can continue to process the matrix operation results, thereby maintaining the normal operation of the large language model as a whole.

[0072] Step S144: Determine the query result corresponding to the input data according to the output result of the floating-point data type.

[0073] For example, the floating-point data type may be converted into text to obtain corresponding text, and the query result corresponding to the input data may be determined based on the obtained text.

[0074] The following further explains the process of calculating the weight parameters of the linear calculation layer and the integer query vector (ie, the process of steps S141 to S143 ) through a specific example.

[0075] As shown in Figure 5, when the above-mentioned W4A8 quantization strategy is used for model calculation, the weight parameters of the linear calculation layer are quantized to 4 bits, and the input is a floating-point query vector (fp16 is used as an example here). During calculation, the weight parameters of the linear calculation layer quantized to 4 bits (i.e., int4) are read from the memory, and the scale and zero information of the weight parameters of the linear calculation layer when quantized are obtained; the int4 type weight parameters of the linear calculation layer are converted to int8 type weight parameters, and the floating-point query vector (i.e., the input vector in Figure 5, which is fp16 or fp32 floating-point type) is quantized to an int8 integer query vector. Then, a matrix multiplication operation is performed on the int8 integer query vector and the int8 type weight parameter to obtain an int32 matrix operation result. The scaling factor when the floating-point query vector is quantized and the scale and zero information when the weight parameter is quantized are used to dequantize the int32 matrix operation result to obtain a floating-point output result of the floating-point data type.

[0076] Taking the W4A8 quantization strategy as an example, floating-point query vectors typically need to pass through multiple decoder layers of a large language model. For example, in a 7B-parameter LLM model, each layer performs linear computations (matrix multiplications) on the vectors. Assuming the input floating-point query vector is a and the trained decoder weight parameter is w, the matrix multiplication a@w is required, where @ represents matrix multiplication. If this process is performed using FP32 (i.e., both a and w are in FP32 format), the CPU must read at least w data from memory for each decoder layer, requiring 28GB of memory access per round. Using FP16 format requires at least 14GB of memory access. However, using the W4A8 quantization strategy, where w is 4 bits, requires 3.5GB of memory access per round, significantly reducing total memory accesses. Using 8-bit a also reduces memory accesses while leveraging int8 computation instructions, significantly improving computational efficiency.

[0077] In one embodiment, before step S140 , the following steps S140 a to S140 c may be further included.

[0078] Step S140a: Obtain the single calculation dimension size corresponding to the calculation instruction supported by the user terminal.

[0079] For example, when the user terminal supports the signed integer matrix multiply-accumulate (SMMLA) instruction, the specific operation process of the SMMLA instruction is [2,8]@[2,8]→[2,2]. Therefore, the single calculation dimension size corresponding to the SMMLA instruction is 8. When the user terminal supports the signed dot product (SDOT) instruction, the single calculation dimension size corresponding to the SDOT instruction is 4.

[0080] Step S140b: Determine a data block size that matches the above-mentioned single calculation dimension size.

[0081] For example, when the size of the single calculation dimension is 8, the size of the data block is also 8; when the size of the single calculation dimension is 4, the size of the data block is also 4.

[0082] Step S140c: rearrange the data arrangement of the integer query vector and the weight parameters of the linear calculation layer according to the data block size to obtain a rearranged query vector and a rearranged weight parameter.

[0083] Specifically, step S140c obtains a rearranged query vector and a rearranged weight parameter, the size of a single calculation dimension of which is the same as the size of a single calculation dimension corresponding to the above calculation instruction.

[0084] In a specific embodiment, the data arrangement shape of the integer query vector is a two-dimensional matrix consisting of the batch size of the integer query vector and the number of channels of the input data of the large language model, and the data arrangement shape of the weight parameters of the linear calculation layer is a two-dimensional matrix consisting of the number of channels of the output feature map in the weight parameters of the linear calculation layer and the number of channels of the input feature map in the weight parameters of the linear calculation layer. Specifically, the data arrangement shape of the above-mentioned integer query vector can be [batch, ic], and the data arrangement shape of the weight parameters of the above-mentioned linear calculation layer is [oc, ic], where batch represents the integer query vector. Batch size, that is, the number of division units corresponding to the input data. For example, if the input data is divided into 5 word segmentation units or 5 characters, the batch is 5. The ic corresponding to the integer query vector represents the number of channels of the input data of the large language model (that is, the number of feature dimensions), that is, the number of dimensions of the features included in the input data. For example, if the input data includes features of four dimensions such as time, place, completed projects, and age, the ic corresponding to the integer query vector is 4; the ic corresponding to the weight parameter represents the number of channels of the input feature map in the weight parameter of the linear calculation layer, and oc represents the number of channels of the output feature map in the weight parameter of the linear calculation layer.

[0085] Correspondingly, the above-mentioned step S1401c can be implemented according to the following steps: the number of channels of the input data of the large language model of the integer query vector is rearranged in blocks according to the data block size to obtain the rearranged query vector; the number of channels of the output feature map of the weight parameters of the linear calculation layer and the number of channels of the input feature map of the linear calculation layer are rearranged in blocks according to the data block size to obtain the rearranged weight parameters.

[0086] Specifically, step S1401c can be implemented according to the following step S140c-1.

[0087] Step S1410c-1: Rearrange the data arrangement shape of the integer query vector to [ic / pack, batch, pack], and rearrange the data arrangement shape of the weight parameters of the linear calculation layer to [oc / pack, ic / pack, pack, pack].

[0088] Wherein, pack represents the size of the data block.

[0089] For example, the weight parameter size of a Linear layer is [4096, 4096]. If the data block size is 8, the weight parameter will be rearranged to [512, 512, 8, 8]. If the data block size is 4, the weight parameter will be rearranged to [1024, 1024, 4, 4].

[0090] For example, when the user asks "Please synthesize a landscape picture", 14 tokens can be generated through the tokenizer layer, that is, 14 partitioning units are generated. These 14 tokens can be converted into a vector of size [14, 4096] through the embedding layer. Assuming that the instruction set supported by the user's mobile phone includes instructions corresponding to the block data size of 8, for example, the user's mobile phone supports the smmla instruction, then pack = 8, and the input vector can be rearranged to [512, 14, 8]; the weight parameters of the Linear layer have been pre-arranged to [512, 512, 8, 8]; therefore This calculation logic will be executed according to [512,512,8,8]@[512,14,8], where the first dimension 512 in [512,512,8,8] is the dimension that can be multi-threaded in parallel, the 512 in [512,14,8] and the second dimension 512 in [512,512,8,8] are the dimensions of loop calculation, the 14 in [512,14,8] and the third dimension 8 in [512,512,8,8] are the output dimensions; the 8 in [512,14,8] and the fourth dimension 8 in [512,512,8,8] are the dimensions when using Single Instruction Multiple Data (SIMD) instructions for calculation.

[0091] Correspondingly, step S140 can be implemented according to the following steps: using the batch size of the integer query vector and the data block size as the output dimension, the data block size as the calculation dimension during a single calculation of the calculation instruction, and using the calculation instruction to operate the rearranged query vector and the rearranged weight parameter to obtain the query result corresponding to the input data.

[0092] Specifically, step S140 can be implemented according to the following step S145.

[0093] Step S145: Calculate the rearranged query vector and the rearranged weight parameter using the calculation instruction supported by the user terminal to obtain a query result corresponding to the input data.

[0094] Specifically, step S145 can run the calculation of the corresponding dimensions of oc / pack in parallel, and cyclically calculate the calculation of the corresponding dimensions of ic / pack in a cyclic calculation manner, with the batch and the pack as the output dimensions, and the pack as the calculation dimension for a single calculation of the calculation instruction. The calculation instruction is used to operate the rearranged query vector and the rearranged weight parameter to obtain the query result corresponding to the input data.

[0095] After the weight parameters of the Linear layer and the integer query vector corresponding to the input data are rearranged using the solution provided by this embodiment, for example, [4096,4096]@[14,4096] is rearranged to [512,512,8,8]@[512,14,8]. When the smmla instruction is used to perform matrix operations, the specific operation of this hardware instruction is [2,8]@[2,8]->[2,2]. After the data is rearranged by this embodiment, the calculation logic in the data loop is [8,8]@[14,4096]. [14,8]->[8,14]. Since the single calculation amount in the calculation logic within the loop after rearrangement is 8, which is the same as the single calculation amount of the SMMLA instruction, the logical operation [8,8]@[14,8]->[8,14] can be directly completed through multiple SMMLA instructions, which is more convenient for the user terminal to perform data operations and improves the calculation efficiency. At the same time, the data is read and written continuously, with good data continuity and more compact data arrangement, so that the calculation instructions can perform more compact matrix multiplication operations, further improving the calculation speed.

[0096] In a specific embodiment, step S145 can be implemented by following steps S145a to S145b.

[0097] Step S145a: Obtain the available number of registers corresponding to the user terminal.

[0098] Step S145b: Determine the number of cycles for the calculation instruction to be performed based on the available number of registers and the batch size of the integer query vector, and based on the number of cycles, perform operations on the rearranged query vector and the rearranged weight parameter through the calculation instruction to obtain the query result corresponding to the input data.

[0099] When using assembly language to implement a computing kernel (core function), that is, when using assembly language to generate a computing function for the underlying computing logic that supports a large language model, it is necessary to consider the availability of the number of registers in the processor architecture of the user terminal (such as the ARM architecture) to make full use of the available registers to improve computing efficiency.

[0100] Specifically, taking the example mentioned above as an example, the number of input tokens is assumed to be batch, then the calculation scale is [512,512,8,8]@[512,batch,8], and the underlying calculation logic (i.e. kernel) that needs to be implemented is: [8,8]@[bacth,8]->[8,bacth]. If this kernel is implemented directly without a loop, at least 5+batch registers are required. Depending on the number of available registers (for example, arm64 has 32 registers), calculations with batches smaller than 27 can be implemented. For example, the calculation logic for batches of 12, 10, 8, 4, 2, 1, etc. can be calculated without setting When the number of available registers is 10, at least 5+batch registers are required. Therefore, when the batch is less than or equal to 5, no loop is required. When the batch is greater than 5, a loop is required to implement the underlying calculation logic [8,8]@[bacth,8]->[8,bacth]. For example, if the batch is greater than 5 and less than or equal to 10, the number of loops is 2. When the batch is greater than 10 and less than or equal to 15, the number of loops is 3. Therefore, the number of loops should be determined in such a way that more registers are used in a single kernel to calculate more data, thereby reducing memory access and improving the computational efficiency of large language models.

[0101] When performing calculations in step S145b, the calculation instruction may loop the calculation task of operating the rearranged query vector and the rearranged weight parameter for the above-mentioned number of loops, for example, loop the calculation task of [8,8]@[bacth,8]->[8,bacth] for the above-mentioned number of loops to obtain the query result corresponding to the input data.

[0102] This embodiment determines the number of loops according to the number of available registers in the user terminal, so as to utilize more available registers for calculation in a single calculation, thereby further reducing the amount of memory access.

[0103] In one implementation, step S145 may be implemented as follows: step S145c to step S145d.

[0104] Step S145c: Determine available threads in the user terminal.

[0105] Step S145d: When the number of available threads is less than the parallel dimension threshold, the calculation instruction is used to calculate the rearranged query vector and the rearranged weight parameter in parallel through each available thread.

[0106] For example, since the first dimension corresponding to the weight parameter can be parallelized after the data shape is rearranged, the work can be divided according to the number of threads in this dimension for parallel calculation. For example, in the example above, for the calculation task [512,512,8,8]@[512,14,8], the first dimension in [512,512,8,8] is 512. Assuming that 4 threads are used for calculation, each thread can perform 128 data calculations, that is, each thread is responsible for the calculation task of [128,512,8,8]@[512,14,8]. This division method is to divide the parallel tasks in an equal manner. Optionally, each parallel calculation task can also be divided in an uneven manner. For example, when it is not possible to divide it evenly, the excess part can be put into the last thread for execution.

[0107] This embodiment adopts a multi-threaded parallel computing approach, which can fully utilize the multi-core computing capabilities of the user terminal, improve multi-core performance, and thus improve the computing efficiency of large language models.

[0108] In one embodiment, the processing layers of the above-mentioned large language model other than the embedding layer can be loaded into the memory of the user terminal so that the large language model can perform information reasoning through the memory. The processing layers other than the embedding layer may include a text conversion tokenizer layer, a linear calculation Linear layer, an attention layer, an activation function layer, a loss function layer, etc., and may also include other data processing layers other than the embedding layer. The above-mentioned embedding layer can be deployed on the disk of the user terminal. For example, the model file corresponding to the embedding layer can be saved on the disk of the user terminal. The embedding layer includes the embedding layer weight parameters corresponding to each data.

[0109] Correspondingly, the above step S120 can be implemented according to the following steps S121 to S122.

[0110] Step S121: determining a target weight parameter corresponding to the input data from the weight parameters of each embedding layer included in the embedding layer, and loading the target weight parameter into the memory of the user terminal.

[0111] Step S122: Convert the input data into a floating-point query vector of the corresponding floating-point data type using the target weight parameter.

[0112] When it is detected that the process of determining the corresponding vector of the input data through the embedding layer is completed, the target weight parameter is deleted from the memory to reduce memory usage.

[0113] For example, if the embedding layer weight corresponding to the embedding layer is 151936×4096 floating-point numbers, the embedding layer weight parameters corresponding to the token ID can be selected based on the word segmentation unit identification sequence (token ID sequence, ID range is 0-151936) input to the embedding layer. Each ID corresponds to an embedding layer weight of 4096 floating-point numbers. The general implementation in the related art loads the entire embedding layer weight into memory and obtains the embedding layer weight parameters at the corresponding position based on the ID sequence. This method consumes a large amount of user terminal memory and reduces the operating efficiency of the user terminal.

[0114] In the embodiment of the present application, since word embedding is implemented by selecting corresponding data from N according to id from the N×H embedding layer weight parameters, for example, if an input sequence is [2,9886,32], it is only necessary to load the floating-point numbers corresponding to the three ids 2, 9886, and 32 from the disk according to the offsets corresponding to these ids, that is, to load 3×4096 floating-point numbers, without having to load 151936×4096 floating-point numbers. Therefore, it can greatly reduce memory usage and improve the computational efficiency of large language models.

[0115] Optionally, the embedding layer weight parameters corresponding to the embedding layer can be stored using the 16-floating-point number bf16 data type, which can reduce the amount of calculation by half compared to the original embedding layer weight parameters.

[0116] This embodiment stores the embedding layer weight parameters on disk without loading all of them into memory, and loads the required parts into memory by reading files to implement word embedding, which can improve the operating efficiency of the user terminal.

[0117] In a specific embodiment, in step S122 , the input data may be converted into a floating-point query vector of a corresponding floating-point data type according to the following steps S122 a to S122 b .

[0118] Step S122a: The text conversion layer based on the large language model converts the input data into converted data corresponding to the data format supported by the large language model.

[0119] Specifically, the input data can be divided into various division units based on the tokenizer layer of the large language model, and then the division units are mapped to the vocabulary corresponding to the data format supported by the large language model. The vocabulary includes identifiers corresponding to each division unit, and the identifier token corresponding to the division unit corresponding to the input data is obtained. The identifier token corresponding to the division unit is the above-mentioned converted data.

[0120] Step S122b: Convert the converted data into a floating-point query vector of the corresponding floating-point data type using the target weight parameter.

[0121] This embodiment converts input data into a data format supported by the large language model through a text conversion layer, and then performs corresponding vector conversion on the converted data, thereby enabling the large language model to infer different types of input data, thereby increasing the number of language types supported by the large language model and making it more universally applicable.

[0122] In one embodiment, before step S110, the above method may further include the following steps S110a to S110b.

[0123] Step S110a: Obtain a model file corresponding to the large language model for deployment on a user terminal. The model file includes text conversion layer files, linear calculation layer files, and embedding layer files. Each text conversion layer file is represented by a file in a unified text format.

[0124] The model file used for deployment on the user terminal can be a mobile neural network (MNN) type file, or other model files used for deployment on the user terminal. The model file used for deployment on the user terminal is more suitable for deploying large language models in mobile user terminals such as mobile phones and tablets, thereby improving the convenience of model deployment.

[0125] Because different large language models (LLMs) use different tokenizer models in different formats, including text-based (such as the tiktoken model), JSON-based (such as the GPT2Tokenizer model), and protobuf-based (such as the sentencepiece model), directly supporting these tokenizers on the client side requires support for multiple file formats and a lot of code logic to implement, which increases the size of the binary file during final deployment. Exporting them to a unified text file in txt format can reduce the complexity of the code required to implement the large language model during deployment and the binary size corresponding to the tokenizer processing logic, making the large language model more lightweight and easier to deploy and run on user terminals.

[0126] Step S110b: loading the text conversion layer file and the linear calculation layer file into the user terminal memory, and deploying the embedding layer file into the disk of the user terminal.

[0127] Through the solution of this embodiment, a large language model can be deployed on a user terminal in a lightweight manner.

[0128] An embodiment of the present application also provides a method for exporting a large language model, which can export the above-mentioned model file. The export method includes the following steps A to E.

[0129] Step A: Divide the large language model into blocks based on the different functions implemented to obtain model blocks corresponding to each function, wherein the model blocks include each embedding layer block and each linear calculation layer block.

[0130] Optionally, the above-mentioned model blocks may also include blocks for each text conversion layer. Step A abstracts the large language model into multiple substructures for block export, which can reduce the size of a single model and facilitate distribution.

[0131] Step B: Export the above linear calculation layer blocks and the above embedding layer blocks into a model file format that can be recognized by the user terminal, and obtain the user terminal embedding layer blocks and the user terminal linear calculation layer blocks.

[0132] For example, the linear computation layer blocks and the embedding layer blocks can be exported as the MNN model files. In the embodiment of the present application, when exporting the model, the large language model can be first exported as a general ONNX model, and then the ONNX model can be converted into an MNN model using a conversion tool such as mnnconvert. During the conversion process, the MNN model can be quantized into a model encoding method of data types such as 8-bit, 4-bit, bf16, and pf16.

[0133] Step C: exporting each text conversion layer block into a block file in a unified text format to obtain text conversion layer blocks in a text format.

[0134] In the process of exporting each text conversion layer block into a block file in a unified text format, the special representation symbols in the text conversion layer can be converted into text corresponding to the original meaning of the special representation symbols, and the text conversion layer block can be saved using base64 encoding.

[0135] Step D: quantizing the weight parameters of the linear calculation layer block of the user terminal into format data of an integer data type to obtain a quantized linear calculation layer block.

[0136] Step E: Determine the user terminal embedding layer block, the quantized linear calculation layer block, and the text conversion layer block in the text format as the model file corresponding to the large language model for deployment in the user terminal.

[0137] Steps A through E may be performed by the aforementioned user terminal. In this case, a communication connection may be established between the user terminal and the server that deploys the original large language model to download and export the model file for deployment on the user terminal from the server. Steps A through E may also be performed by the server that deploys the original large language model, but this application does not specifically limit this.

[0138] Optionally, the above step A can be implemented according to the following steps a to c.

[0139] Step a: Divide the large language model into blocks based on the different functions implemented to obtain the model blocks to be optimized corresponding to each function.

[0140] Step b: Replace the code related to the data shape information in the code corresponding to the above-mentioned model block to be optimized with the operator code that is not directly related to the data shape information to obtain the replaced block.

[0141] Large language models may have dynamic shape issues during export. This refers to the situation where, due to code implementation issues, only the exported shape is supported when exporting a large language model to the ONNX format. This can lead to errors when the input shape changes during actual runtime. To address this, some code related to shape needs to be modified. Specifically, the code related to data shape information in the code corresponding to the model blocks to be optimized is replaced with operator code that is not directly related to data shape information.

[0142] For example, some operators related to the shape of the tensor can be combined and replaced by operators such as squeeze, unsqueeze, and transpose that are not directly related to the shape information.

[0143] Step c: Delete the redundant data in the replaced blocks to obtain the model blocks corresponding to each function.

[0144] Specifically, the calculation graph in the replaced block can be operated on by merging operators and / or deleting redundant operators, the position constant encoding in the replaced block can be deleted, and other redundant data can also be deleted.

[0145] Positional constant encoding mainly refers to the fact that many LLM positional encoding implementations will calculate a positional encoding cache according to the maximum length. When exported to ONNX, it will cause each model block to generate a large constant (tens of megabytes). This part of the data can be calculated according to the actual input length, and then this part of the positional constant encoding can be deleted.

[0146] In a specific embodiment, when exporting a large language model, as shown in FIG4 , the export of the large language model can be implemented through the following functional modules.

[0147] As shown in Figure 4, the code rewriting module coderefactor can be used to rewrite the PyTorch code corresponding to the original large language model and correct the problems mentioned above, such as dynamic shape and redundant data. The block module classabstract can be used to abstract each model block to achieve unified calculation and export logic for each model block. The hierarchical module blocksplit can be used to split and export the decode block of a multi-layer LLM according to the hierarchical information of the LLM. For example, an N-layer decode block can be exported as N block models respectively, the norm layer of the LLM is added to the last block model, and the linear transformation layer and sampling function (such as argmax) of the LLM are merged together to export the lm layer. Since ONNX stores additional weights when the file is larger than 2GB, resulting in the separation of the computation graph and weights, this embodiment exports them separately to avoid the problem of separation of the computation graph and weights, while reducing the size of a single file for easy distribution; the model rewrite module tokenizer rewrite can rewrite various tokenizer models into a unified txt format, while converting some special symbols to their original meaning and saving them in base64 encoding; the export module onnx export can export the entire LLM model one by one according to the abstract structure, or export them together; the conversion module mnnconvert can convert the onnx model to an mnn model, and the graph optimization module graph optimize can perform computation graph optimization; the weight quantization module weight quantizes the model's floating-point weight parameters into 4-bit or 8-bit weight parameters. Based on the processing logic executed by each of the above functional modules, a unified text format tokenizer txt corresponding to the MNN model and the tokenizer layer for deployment on the mobile terminal can be obtained. Since the execution process of each functional module for model export in Figure 4 has been introduced above, the specific execution method of each functional module in the process of exporting a large language model will not be described in detail here.

[0148] When using the solution provided in the embodiment of the present application to run a large language model on a user terminal, it is necessary to first convert the large language model into a unified text file corresponding to the above-mentioned tokenizer (referred to as tokenizer text) and the above-mentioned mnn model file. When the large language model is running, it is necessary to load the files in the mnn model file except the embedding layer file into the memory, and store the embedding layer file on the disk of the user terminal; the reasoning framework of the user terminal is based on MNN, and the MNN model framework can load the model and obtain output based on calculation.

[0149] When a user terminal uses the solution provided in an embodiment of the present application to query data through a large language model, the user terminal stores the converted tokenizer file and the mnn model file. When the large language model is started, these two files are loaded into the memory. The embedding part in the mnn model file does not need to be loaded, and the embedding can be stored on the disk.

[0150] Exemplarily, after the user enters a text on the interface of the large language model of the user terminal to perform data query, the user terminal can perform data inference query through the following steps 1 to 3.

[0151] Step 1: Convert it into an integer array through tokenizer, and use Embedding to convert the integer array into a vector;

[0152] Step 2: The vector from step 1 is used as the input of the MNN model corresponding to the decoder block of the decoding layer. The MNN performs inference on the model and obtains the output. The output is used as the input of the next block to perform inference, and all the processing layers of the model are executed in sequence. The output of the last layer is passed to the LM layer for inference to obtain the token ID of the next word.

[0153] Step 3: Convert the token ID into text through the tokenizer. If streaming output is used, the text is displayed in the generated content. The current token ID continues to obtain the next text through the process of steps 1 and 2 until the end text is encountered. The process is terminated to obtain the query result corresponding to the text entered by the user.

[0154] The method for processing data based on a large language model provided in an embodiment of the present application is applied to a user terminal. A large language model is deployed on the user terminal, and the weight parameters of each linear calculation layer of the large language model are pre-quantized into data in the format of an integer data type. Since the weight parameters of each linear calculation layer are quantized into integer data types, compared with weight parameters of floating-point data types, the weight parameters of integer data types can significantly reduce the amount of memory access when performing matrix operations. Since the memory of the user terminal is usually relatively small, using weight parameters of integer data types to perform matrix operations on input data can significantly improve the model's calculation and memory access efficiency, thereby improving the calculation efficiency of the large language model and the requirements for device resources, allowing the user terminal to run the large language model more smoothly. The weight parameters of integer data types can also effectively replace the information of weight parameters of floating-point data types, ensuring calculation accuracy. When processing data through a large language model, the solution provided in the present application first performs vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data, and then converts the floating-point query vector into an integer query vector of an integer data type. The integer query vector is operated on by the weight parameters of the linear calculation layer to obtain a query result corresponding to the input data. When the integer query vector of the integer data type of the input data performs operations such as matrix multiplication in the linear calculation layer, the memory access amount can be further reduced, thereby improving the computing efficiency of the user terminal.

[0155] It can be seen that the solution provided by the present application deploys a large language model on a user terminal for operation. By quantizing the weight parameters of the large language model and the input data entered by the user into data of integer data types, the amount of memory access and the amount of calculation required by the large language model when running on the user terminal can be greatly reduced, making it more suitable for situations where the user terminal has small memory and limited computing resources, so that the user terminal can run the large language model in the streaming field. In addition, since the distribution difference of the weight parameters of the linear calculation layer of the large language model is usually very small, the distribution similarity of each weight parameter after quantization into an integer type is very high. Therefore, the quantization of the weight parameter has little effect on the calculation accuracy of the large language model, and since the integer data type usually has a sufficient range to represent the input data of the large language model, the query vector corresponding to the input data can be quantized very completely through the integer data type, thereby better ensuring that the information of the input data is not lost. Therefore, the weight parameter quantization and query vector quantization performed in this application can well ensure the computational accuracy of the large language model running on the user terminal. In this way, since the large language model is deployed on the user terminal, the user terminal can reply to the user's query content without being connected to the Internet, which better ensures the privacy of the user's query content and is more convenient for users to use. In addition, since there is no need to equip a dedicated cloud server to deploy the large language model, the operating cost of the large language model can be reduced.

[0156] Example 2

[0157] The second embodiment of the present application also provides a device for processing data based on a large language model corresponding to the data access method embodiment provided in the first embodiment. The device for processing data based on a large language model is applied to a user terminal, on which a large language model is deployed. The weight parameters of each linear calculation layer of the large language model are pre-quantized into data in the format of an integer data type. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For the details of the relevant technical features and the effects achieved, please refer to the corresponding description of the method embodiment for processing data based on a large language model provided above. As shown in Figure 6, the device for processing data based on a large language model provided in this embodiment includes:

[0158] An acquisition unit 201 is used to acquire input data;

[0159] A vector conversion unit 202 is configured to perform vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data; and convert the floating-point query vector into an integer query vector of an integer data type;

[0160] The operation unit 203 is configured to perform an operation on the integer query vector using the weight parameters of the linear calculation layer to obtain a query result corresponding to the input data.

[0161] Optionally, the weight parameters of the linear calculation layer are pre-quantized into 4-bit binary integer int4 format data;

[0162] The vector conversion unit is specifically configured to convert the floating-point query vector into an integer query vector in an 8-bit binary integer int8 format.

[0163] Optionally, the operation unit is specifically used to: convert the weight parameters of the linear calculation layer into weight parameters of a target integer type, where the target integer type is the same as the integer type corresponding to the integer query vector; perform matrix multiplication on the integer query vector and the weight parameters of the target integer type to obtain a matrix operation result; perform dequantization on the matrix operation result to obtain an output result of a floating-point data type; and determine the query result corresponding to the input data based on the output result of the floating-point data type.

[0164] Optionally, the operation unit is specifically used to: obtain the scaling factor and zero point when the weight parameters of the linear calculation layer are quantized, the scaling factor is the scaling factor that maps the floating-point number to the integer range, and the zero point is the offset that maps the zero value of the floating-point number to the zero value of the integer; according to the scaling factor and the zero point, the matrix operation result is dequantized to obtain an output result of the floating-point data type.

[0165] Optionally, the above device may further include:

[0166] a rearrangement unit, configured to obtain a single calculation dimension size corresponding to a calculation instruction supported by the user terminal; determine a data block size that matches the single calculation dimension size; and rearrange the data arrangement shape of the integer query vector and the weight parameters of the linear calculation layer according to the data block size to obtain a rearranged query vector and a rearranged weight parameter;

[0167] The operation unit is specifically configured to: perform operation on the rearranged query vector and the rearranged weight parameter through the calculation instruction to obtain a query result corresponding to the input data.

[0168] Optionally, the data arrangement shape of the integer query vector is [batch, ic], and the data arrangement shape of the weight parameters of the linear calculation layer is [oc, ic], wherein batch represents the batch size of the integer query vector, ic corresponding to the integer query vector represents the number of channels of the input data of the large language model, ic corresponding to the weight parameter represents the number of channels of the input feature map in the weight parameter of the linear calculation layer, and oc represents the number of channels of the output feature map in the weight parameter of the linear calculation layer;

[0169] The rearrangement unit is specifically configured to rearrange the data arrangement shape of the integer query vector into [ic / pack, batch, pack], and rearrange the data arrangement shape of the weight parameters of the linear calculation layer into [oc / pack, ic / pack, pack, pack], where pack represents the data block size;

[0170] The operation unit is specifically used to: run the calculation of the corresponding dimensions of the oc / pack in parallel, cyclically calculate the calculation of the corresponding dimensions of the ic / pack in a cyclic calculation manner, use the batch and the pack as the output dimensions, and use the pack as the calculation dimension for a single calculation of the calculation instruction, use the calculation instruction to operate the rearranged query vector and the rearranged weight parameter to obtain the query result corresponding to the input data.

[0171] Optionally, the operation unit is specifically used to: obtain the number of available registers corresponding to the user terminal; determine the number of cycles for the calculation instruction to perform calculations based on the number of available registers and the batch size of the integer query vector, and based on the number of cycles, perform the operation on the rearranged query vector and the rearranged weight parameter through the calculation instruction to obtain the query result corresponding to the input data.

[0172] Optionally, the computing unit is specifically configured to: determine available threads in the user terminal; and use the computing instruction to compute the rearranged query vector and the rearranged weight parameter in parallel through each of the available threads.

[0173] Optionally, the processing layers of the large language model except the embedding layer are loaded into the memory of the user terminal, the embedding layer is deployed in the disk of the user terminal, and the embedding layer includes embedding layer weight parameters corresponding to each data respectively;

[0174] The vector conversion unit is specifically configured to: determine a target weight parameter corresponding to the input data from each weight parameter included in the embedding layer, and load the target weight parameter into the memory;

[0175] The input data is converted into a floating-point query vector of a corresponding floating-point data type using the target weight parameter.

[0176] Optionally, the vector conversion unit is specifically used to: convert the input data into converted data corresponding to the data format supported by the large language model based on the text conversion layer of the large language model; and convert the converted data into a floating-point query vector of the corresponding floating-point data type through the target weight parameter.

[0177] Optionally, the device further comprises:

[0178] A model deployment unit is used to obtain a model file corresponding to the large language model for deployment on a user terminal, wherein the model file includes text conversion layer files, linear calculation layer files, and embedding layer files, and each text conversion layer file is represented by a file in a unified text format; the text conversion layer file and the linear calculation layer file are loaded into the memory of the user terminal, and the embedding layer file is deployed on the disk of the user terminal.

[0179] Optionally, the device further comprises:

[0180] An export unit is used to divide the large language model into blocks based on the different functions implemented to obtain model blocks corresponding to each function, wherein the model blocks include each embedding layer block, each text conversion layer block, and each linear calculation layer block; the linear calculation layer block and the embedding layer block are exported into a model file format that can be recognized by the user terminal to obtain a user terminal embedding layer block and a user terminal linear calculation layer block; each of the text conversion layer blocks is exported into a block file in a unified text format to obtain a text conversion layer block in a text format; the user terminal embedding layer block, the user terminal linear calculation layer block, and the text conversion layer block in the text format are determined as the model file corresponding to the large language model for deployment on the user terminal.

[0181] Optionally, the export unit is specifically used to: divide the large language model into blocks based on the different functions implemented to obtain model blocks to be optimized corresponding to each function; replace the code about data shape information in the code corresponding to the model block to be optimized with operator code that is not directly related to the data shape information to obtain replaced blocks; delete redundant data in the replaced blocks to obtain model blocks corresponding to each function.

[0182] Optionally, the deriving unit is specifically configured to perform at least one of the following:

[0183] Merging operators and / or deleting redundant operators in the computation graph in the replaced block;

[0184] The position constant code in the replaced block is deleted.

[0185] The third embodiment of the present application also provides an electronic device embodiment corresponding to the method for processing data based on a large language model provided in the first embodiment. The electronic device is a user terminal, on which a large language model is deployed. The weight parameters of each linear calculation layer of the large language model are pre-quantized into integer data format. The following description of the electronic device embodiment is merely illustrative. The electronic device embodiment is as follows:

[0186] Please refer to Figure 7 to understand the above electronic device, which is a schematic diagram of the electronic device. The electronic device provided in this embodiment includes: a processor 1001, a memory 1002, a communication bus 1003, and a communication interface 1004;

[0187] The memory 1002 is used to store computer instructions for data processing. When the computer instructions are read and executed by the processor 1001, the following steps are performed:

[0188] Get input data;

[0189] Performing vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data;

[0190] Converting the floating-point query vector to an integer query vector of an integer data type;

[0191] The query result corresponding to the input data is obtained by performing an operation on the weight parameter of the linear calculation layer and the integer query vector.

[0192] The fourth embodiment of the present application also provides a computer-readable storage medium for implementing the method described in the first embodiment. The computer-readable storage medium embodiment provided in this application is described in a relatively simple manner. For relevant parts, please refer to the corresponding description of the above method embodiment. The embodiment described below is merely illustrative.

[0193] The computer-readable storage medium provided in this embodiment stores computer instructions. The computer-readable storage medium can be applied to a user terminal. A large language model is deployed on the user terminal. The weight parameters of each linear calculation layer of the large language model are pre-quantized into integer data format. When the instructions are executed by a processor, the following steps are implemented:

[0194] Get input data;

[0195] Performing vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data;

[0196] Converting the floating-point query vector to an integer query vector of an integer data type;

[0197] The query result corresponding to the input data is obtained by performing an operation on the weight parameter of the linear calculation layer and the integer query vector.

[0198] The fifth embodiment of the present application also provides a computer program product for implementing the method described in the first embodiment. The computer program product embodiment provided in this application is described relatively simply. For relevant parts, please refer to the corresponding description of the above method embodiment. The embodiment described below is only illustrative.

[0199] The computer program product provided in this embodiment includes a computer program, which, when executed by a processor, implements the following steps:

[0200] Get input data;

[0201] Performing vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data;

[0202] Converting the floating-point query vector to an integer query vector of an integer data type;

[0203] The query result corresponding to the input data is obtained by performing an operation on the weight parameter of the linear calculation layer and the integer query vector.

[0204] The sixth embodiment of the present application further provides a large language model, which includes:

[0205] A text conversion tokenizer layer, used to convert input data into a text data format supported by the large language model;

[0206] Embedding layer, used to convert text data into vectors;

[0207] The Transformer layer includes various linear calculation Linear layers, the weight parameters of the Linear layers are quantized into integer data type format data, and the Linear layers are used to perform linear transformation on the input vector data to obtain the model operation results.

[0208] The various computing layers of the large language model provided in this embodiment have been described in detail in the first embodiment and will not be described in detail here.

[0209] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0210] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0211] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0212] 2. Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0213] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A method for processing data based on a large language model, characterized in that: The method is applied to a user terminal, wherein a large language model is deployed on the user terminal, and weight parameters of each linear calculation layer of the large language model are pre-quantized into data in the format of an integer data type; the method comprises: Get input data; Performing vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data; Converting the floating-point query vector to an integer query vector of an integer data type; The query result corresponding to the input data is obtained by performing an operation on the weight parameter of the linear calculation layer and the integer query vector.

2. The method according to claim 1, characterized in that The operation of the weight parameter of the linear calculation layer and the integer query vector to obtain a query result corresponding to the input data includes: Converting the weight parameters of the linear calculation layer into weight parameters of a target integer type, where the target integer type is the same as the integer type corresponding to the integer query vector; Performing matrix multiplication on the integer query vector and the weight parameter of the target integer type to obtain a matrix operation result; Dequantizing the matrix operation result to obtain an output result of floating-point data type; The query result corresponding to the input data is determined according to the output result of the floating-point data type.

3. The method according to claim 2, characterized in that Dequantizing the matrix operation result to obtain an output result of a floating-point data type includes: Obtaining a scaling factor and a zero point when quantizing a weight parameter of the linear calculation layer, wherein the scaling factor is a scaling factor for mapping a floating-point number to an integer range, and the zero point is an offset for mapping a zero value of a floating-point number to a zero value of an integer; The matrix operation result is dequantized according to the scaling factor and the zero point to obtain an output result of a floating-point data type.

4. The method according to claim 1, wherein Before performing the operation on the integer query vector and the weight parameters of the linear calculation layer, the method further includes: Obtaining a single calculation dimension size corresponding to a calculation instruction supported by the user terminal; Determine a data block size that matches the size of the single calculation dimension; Rearranging the data arrangement of the integer query vector and the weight parameters of the linear calculation layer according to the data block size to obtain a rearranged query vector and a rearranged weight parameter; The operation of the weight parameter of the linear calculation layer and the integer query vector to obtain a query result corresponding to the input data includes: The calculation instruction is used to operate on the rearranged query vector and the rearranged weight parameter to obtain a query result corresponding to the input data.

5. The method according to claim 4, characterized in that The data arrangement shape of the integer query vector is a two-dimensional matrix consisting of the batch size of the integer query vector and the number of channels of the input data of the large language model; the data arrangement shape of the weight parameters of the linear calculation layer is a two-dimensional matrix consisting of the number of channels of the output feature map in the weight parameters of the linear calculation layer and the number of channels of the input feature map in the weight parameters of the linear calculation layer; The step of rearranging the data arrangement of the integer query vector and the weight parameters of the linear calculation layer according to the data block size to obtain the rearranged query vector and the rearranged weight parameters includes: Rearranging the number of channels of the input data of the large language model of the integer query vector in blocks according to the data block size to obtain a rearranged query vector; and rearranging the number of channels of the output feature map of the weight parameter of the linear calculation layer and the number of channels of the input feature map of the linear calculation layer in blocks according to the data block size to obtain rearranged weight parameters; The performing operation on the rearranged query vector and the rearranged weight parameter by the calculation instruction to obtain a query result corresponding to the input data includes: The batch size of the integer query vector and the data block size are used as output dimensions, the data block size is used as the calculation dimension during a single calculation of the calculation instruction, and the calculation instruction is used to operate the rearranged query vector and the rearranged weight parameter to obtain a query result corresponding to the input data.

6. The method according to claim 5, characterized in that The using the calculation instruction to operate the rearranged query vector and the rearranged weight parameter to obtain a query result corresponding to the input data includes: Obtaining the available number of registers corresponding to the user terminal; The number of cycles for the calculation instruction to be performed is determined based on the available number of registers and the batch size of the integer query vector, and the rearranged query vector and the rearranged weight parameter are operated on by the calculation instruction based on the number of cycles to obtain a query result corresponding to the input data.

7. The method according to claim 5, characterized in that The using the calculation instruction to operate the rearranged query vector and the rearranged weight parameter includes: Determining available threads in the user terminal; The calculation instruction is used to calculate the rearranged query vector and the rearranged weight parameter in parallel through each of the available threads.

8. The method according to any one of claims 1 to 7, characterized in that The processing layers of the large language model except the embedding layer are loaded into the memory of the user terminal, the embedding layer is deployed on the disk of the user terminal, and the embedding layer includes embedding layer weight parameters corresponding to each data; The performing vector conversion on the input data through the embedding layer of the large language model to obtain a floating-point query vector of a floating-point data type corresponding to the input data includes: Determining a target weight parameter corresponding to the input data from each weight parameter included in the embedding layer, and loading the target weight parameter into the memory; The input data is converted into a floating-point query vector of a corresponding floating-point data type using the target weight parameter.

9. The method according to claim 8, characterized in that The converting the input data into a floating-point query vector of a corresponding floating-point data type using the target weight parameter includes: Converting the input data into converted data corresponding to a data format supported by the large language model by a text conversion layer based on the large language model; The converted data is converted into a floating-point query vector of a corresponding floating-point data type using the target weight parameter.

10. The method according to claim 9, characterized in that Before obtaining the input data, the method further includes: Obtaining a model file corresponding to the large language model for deployment on a user terminal, the model file including text conversion layer files, linear calculation layer files, and embedding layer files, wherein each text conversion layer file is represented by a file in a unified text format; The text conversion layer file and the linear calculation layer file are loaded into the memory of the user terminal, and the embedding layer file is deployed in the disk of the user terminal.

11. The method according to claim 10, characterized in that The model file is exported in the following way: Dividing the large language model into blocks based on different functions to be implemented to obtain model blocks corresponding to each function, wherein the model blocks include each embedding layer block, each text conversion layer block, and each linear calculation layer block; Exporting the linear calculation layer block and the embedding layer block into a model file format recognizable by the user terminal to obtain a user terminal embedding layer block and a user terminal linear calculation layer block; Exporting each of the text conversion layer blocks into a block file in a unified text format to obtain text conversion layer blocks in a text format; The user terminal embedding layer block, the user terminal linear calculation layer block and the text conversion layer block in the text format are determined as model files corresponding to the large language model for deployment in the user terminal.

12. The method according to claim 11, characterized in that The large language model is divided into blocks based on different functions to obtain model blocks corresponding to each function, including: Dividing the large language model into blocks based on different functions to be implemented, and obtaining model blocks to be optimized corresponding to each function; Replacing the code related to the data shape information in the code corresponding to the block of the model to be optimized with an operator code that is not directly related to the data shape information, to obtain a replaced block; The redundant data in the replaced blocks are deleted to obtain model blocks corresponding to each function.

13. The method according to claim 12, characterized in that The deleting of redundant data in the replaced block includes at least one of the following: Merging operators and / or deleting redundant operators in the computation graph in the replaced block; The position constant code in the replaced block is deleted.

14. A large language model, characterized in that include: A text conversion tokenizer layer, used to convert input data into a text data format supported by the large language model; Embedding layer, used to convert text data into vectors; The Transformer layer includes various linear calculation Linear layers, the weight parameters of the Linear layers are quantized into integer data type format data, and the Linear layers are used to perform linear transformation on the input vector data to obtain the model operation results.

15. An electronic device, characterized in that: include: a processor, a memory, and computer program instructions stored on the memory and executable on the processor; When the processor executes the computer program instructions, the method according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images