Optimizer quantization method, device and controller based on text generation model

Abstract

The present application relates to the field of artificial intelligence technology, and in particular to an optimizer quantization method, device, and controller based on a text generation model. The optimizer quantization method includes reading a text input tensor of an optimizer, where the text input tensor is floating-point data of the first bit width; determining the gradient information of the text input tensor and processing the gradient information in blocks to obtain multiple independent blocks; quantizing the independent blocks according to a normalization constant to obtain quantization results of the independent blocks, where the quantization results are integer data of the second bit width; optimizing the quantization results to obtain optimized quantization results, and using the optimized quantization results as the first optimizer state; performing inverse quantization on the first optimizer state to obtain a second optimizer state, and updating the optimizer; quantizing the second optimizer state to return to the first optimizer state, and storing the optimized quantization results of the independent blocks, which is beneficial to reducing the video memory usage of the optimizer in the text generation model and improving the utilization rate of the graphics card.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an optimizer quantization method, device, and controller based on a text generation model. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, large deep neural network models have been widely used in numerous fields, including chatbots, image generation, video understanding, and text generation. The application of large deep neural network models is currently growing exponentially. Among them, text generation models are essential in dialogue systems, recommendation systems, and search engines. Consequently, the requirements for video memory and quantization for text generation models are becoming increasingly demanding. Currently, training text generation models typically requires storing model parameters, model gradients, and optimizer states in a fixed amount of available memory. The state optimizer maintains gradient statistics over time, using up a significant portion of the memory available for model parameters. This significantly limits the maximum size of trained large deep neural network models.

[0003] In the related art, in order to alleviate the pressure on video memory during the training of text generation models, larger model training is achieved by reducing or effectively allocating the memory required for text generation model parameters. For example, information such as model parameters, model gradients, and optimizer status are chopped up and then distributed to different graphics cards, so that the video memory occupied by each graphics card is evenly shared, so that the same number of cards can train larger text generation models. Although this method of distributing optimizers across multiple cards is effective, it can only be used in the case of multiple graphics cards and needs to be used when data parallelism is used. In addition, optimizer sharding may also generate a large amount of communication overhead, thereby slowing down the training speed. In addition, although some current quantization methods can reduce the memory usage of text generation models, the performance of the text generation model will be reduced during the quantization process, resulting in low convergence of the text generation model and reduced accuracy. Summary of the Invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the embodiments of the present application provide a method, device, and controller for quantizing an optimizer based on a text generation model, which are beneficial for reducing the graphics memory usage of the optimizer in the text generation model, improving the utilization rate of the graphics card, and thus improving the performance of the text generation model.

[0005] In a first aspect, an embodiment of the present application provides an optimizer quantization method based on a text generation model, comprising:

[0006] Read the text input tensor of the optimizer, where the text input tensor is floating-point data with the first bit width;

[0007] Determining gradient information of the text input tensor;

[0008] performing block processing on the gradient information to obtain a plurality of independent blocks, performing normalization processing on the independent blocks to obtain normalization constants of the independent blocks, and performing quantization processing on the independent blocks according to the normalization constant to obtain quantization results of the independent blocks, wherein the quantization results are integer data of a second bit width, wherein the second bit width is smaller than the first bit width;

[0009] Performing optimization preprocessing on the quantization result to obtain an optimized quantization result, and using the optimized quantization result as a first optimizer state;

[0010] performing dequantization processing on the first optimizer state to obtain a second optimizer state, and updating the optimizer according to the second optimizer state;

[0011] The second optimizer state is quantized to return to the first optimizer state, and the optimized quantization result of the independent block is stored.

[0012] According to some embodiments of the present application, the integer data of the second bit width is defined by the following data structure:

[0013] A sign bit, which is the first bit of the data structure;

[0014] exponent bits, used to indicate the size of the exponent bits of the integer data by a number of consecutive zero bits;

[0015] Indication bit, set to 1;

[0016] The linear quantization bit is used to indicate the linear quantization value of the integer data.

[0017] According to some embodiments of the present application, the integer data of the second bit width is defined by the following data structure:

[0018] The fixed bit of the score is the first bit of the data structure;

[0019] exponent bits, used to indicate the size of the exponent bits of the integer data by the number of consecutive zero bits;

[0020] Indication bit, set to 1;

[0021] The linear quantization bit is used to indicate the linear quantization value of the integer data.

[0022] According to some embodiments of the present application, obtaining the quantization result of the independent block includes:

[0023] Converting the text input tensor into a one-dimensional element sequence, and dividing the one-dimensional element sequence into a plurality of independent blocks of a preset interval size;

[0024] Determining a normalization constant for the independent block, and converting the text input tensor to a range of a domain of a target quantized data type according to the normalization constant;

[0025] determining a corresponding value of each element of the one-dimensional sequence of elements in the domain of the target quantized data type;

[0026] Storing the index corresponding to the corresponding value;

[0027] A quantization result of the independent block is obtained according to the index.

[0028] According to some embodiments of the present application, after storing the index corresponding to the corresponding value, the method further includes:

[0029] performing denormalization on the index;

[0030] determining a maximum value of the one-dimensional sequence of elements;

[0031] Determining quantized output data corresponding to the index by a binary search method according to the denormalized index and the maximum value of the one-dimensional element sequence;

[0032] The quantized output data corresponding to the index determined by the binary method is determined as a quantization result of the independent block.

[0033] According to some embodiments of the present application, the text input tensor is input into an embedding layer module to obtain an embedding vector;

[0034] Input the embedding vector into the Transformer model and perform a forward operation to obtain the loss function of the optimizer;

[0035] Performing backpropagation calculation on the loss function to optimize the loss function value;

[0036] Determining the gradient of the optimizer, and obtaining the gradient of the loss function for the text generation model based on the loss function and the gradient;

[0037] The gradient is quantized.

[0038] According to some embodiments of the present application, before inputting the text input tensor into the embedding layer module, the method further includes:

[0039] Normalize the input text input tensor with highly uneven distribution.

[0040] In a second aspect, an embodiment of the present application provides an optimizer quantization device based on a text generation model, comprising:

[0041] A reading module, configured to read a text input tensor of an optimizer, wherein the text input tensor is floating-point data of the first bit width;

[0042] A gradient calculation module, configured to determine gradient information of the text input tensor;

[0043] a block-by-block quantization module, configured to perform block processing on the gradient information to obtain a plurality of independent blocks, and further configured to perform normalization processing on the independent blocks to obtain normalization constants of the independent blocks, and perform quantization processing on the independent blocks according to the normalization constants to obtain quantization results of the independent blocks, wherein the quantization results are integer data of a second bit width, wherein the second bit width is smaller than the first bit width;

[0044] A dynamic quantization module is used to perform optimization preprocessing on the quantization result to obtain an optimized quantization result;

[0045] An embedding layer module, configured to preprocess the optimization quantization result to obtain a first optimizer state;

[0046] The inverse quantization module is used to perform inverse quantization processing on the first optimizer state to obtain a second optimizer state, and update the second optimizer state to the optimizer state; perform quantization processing on the second optimizer state to return to the first optimizer state, and store the quantization result of the independent block.

[0047] In a third aspect, an embodiment of the present application provides a controller comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor runs the computer program, the optimizer quantization method based on the text generation model as described in the technical solution of the first aspect above is executed.

[0048] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the optimizer quantization method based on the text generation model as described in the technical solution of the first aspect.

[0049] The optimizer quantization method, device and controller based on the text generation model provided by the embodiment of the present application have at least one of the following advantages or beneficial effects: reading the text input tensor of the optimizer, wherein the text input tensor is floating-point data of the first bit width; then, calculating the gradient information of the text input tensor, and performing block processing on the gradient information to obtain multiple independent blocks; performing normalization processing on each independent block respectively to solve the normalization constant of each independent block. The gradient information is divided into multiple independent blocks, and each independent block can be normalized separately. There is no need to synchronously solve the gradient information of the input text between the central processing unit cores. Each independent block is processed in parallel across the central processing unit cores, thereby improving the quantization accuracy. The independent block is quantized according to the normalization constant of the independent block to obtain the quantization result of the independent block, wherein the quantization result is integer data of the second bit width. Since each independent block has a corresponding normalization constant, each independent block can be quantized independently. Therefore, the outliers between the independent blocks will not affect each other. The independent blocks can isolate the outliers of other different independent blocks, making the quantization process more accurate. In addition, by quantizing the gradient information of the independent blocks, the outliers can be more evenly distributed across multiple different independent blocks, with smaller absolute quantization errors and relative quantization errors for non-uniform distributions. Optimizing the quantization results of the independent blocks before preprocessing can avoid data overflow problems caused by the narrowing of the data representation range during the quantization process, thereby improving the accuracy of the quantization results of the independent blocks. By dequantizing the first optimizer state to the second optimizer state and performing an update, the performance level of using the second optimizer state can be maintained. Then, the second optimizer state is quantized back to the first optimizer state for storage, which can reduce the video memory usage of the optimizer in the text generation model and improve the utilization of the graphics card. The quantization results of the gradient information of each independent block in the register are used to convert the first optimizer state to the second optimizer state. This eliminates the need to slowly copy the quantization results of multiple independent blocks to the graphics card memory or additional temporary memory to perform quantization and dequantization. This further reduces the graphics card memory usage of the optimizer in the text generation model and improves graphics card resource utilization. In addition, after reading the optimizer's input tensor, the highly unevenly distributed text input tensor is normalized to support more aggressive quantization and avoid extreme gradient changes, thereby improving the stability of the text generation model optimization process.

[0050] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1This is a flowchart of an optimizer quantization method based on a text generation model provided in an embodiment of the present application;

[0052] Figure 2 is a schematic diagram of a data structure of integer data of a second bit width provided in an embodiment of the present application;

[0053] Figure 3 This is a flowchart of a method for obtaining a quantization result of an independent block provided by an embodiment of the present application;

[0054] Figure 4 This is a flowchart of another optimizer quantization method based on a text generation model provided in an embodiment of the present application;

[0055] Figure 5 This is a flowchart of another optimizer quantization method based on a text generation model provided in an embodiment of the present application;

[0056] Figure 6 This is a flowchart of another optimizer quantization method based on a text generation model provided in an embodiment of the present application;

[0057] Figure 7 This is a structural diagram of an optimizer quantization device based on a text generation model provided in an embodiment of the present application;

[0058] Figure 8 This is a schematic diagram of the structure of a controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementation methods. At the same time, the steps or actions in the method description can also be swapped or adjusted in order in a manner that is obvious to those skilled in the art. Therefore, the various orders in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a necessary order, unless otherwise specified that a certain order must be followed.

[0060] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0061] The serial numbers assigned to components herein, such as "first," "second," etc., are used solely to distinguish the objects being described and do not convey any sequential or technical meaning. References to "connection" and "coupling" herein, unless otherwise specified, include both direct and indirect connections (couplings).

[0062] With the rapid development of artificial intelligence (AI), large deep neural network models have been widely used in numerous fields, including chatbots, image generation, video understanding, and text generation. The application of large deep neural network models is currently growing exponentially. Among them, text generation models are essential in dialogue systems, recommendation systems, and search engines. Consequently, the requirements for video memory and quantization for text generation models are becoming increasingly stringent. Currently, when training text generation models, model parameters, model gradients, and optimizer state are typically stored in a fixed amount of available memory. The state optimizer maintains gradient statistics over time, such as exponentially smoothed sums or sums of squares of past gradient values. However, this uses memory allocated to model parameters, causing the optimizer to maintain gradient statistics to consume a significant portion of the overall video memory. This significantly limits the maximum size of large deep neural network models that can be trained.

[0063] In the related art, to alleviate the memory pressure during text generation model training, the memory required for text generation model parameters is reduced or effectively allocated to enable larger model training. For example, information such as model parameters, model gradients, and optimizer status is shredded and distributed across different graphics cards, so that the memory usage of each graphics card is evenly distributed, allowing the same number of graphics cards to train larger text generation models. Although this method of distributing optimizers across multiple graphics cards is effective, it can only be used with multiple graphics cards and requires data parallelism. Optimizer sharding can also generate a large amount of communication overhead, slowing training speed. In addition, although some current quantization methods can reduce the memory usage of text generation models, the quantization process can lead to a decrease in the performance of the text generation model, resulting in low convergence and reduced accuracy. For example, using a 16-bit optimizer, it is difficult to converge and achieve the same model accuracy as a 32-bit optimizer for large text generation models with more than 1B parameters. Therefore, although the above quantization methods reduce memory usage, they lead to a decrease in model performance, and further quantization adjustments are usually required after training.

[0064] Based on this, the embodiments of the present application provide an optimizer quantization method, device and controller based on a text generation model, which is conducive to reducing the graphics memory occupancy of the optimizer in the text generation model, improving the utilization rate of the graphics card, and thus improving the performance of the text generation model.

[0065] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0066] Reference Figure 1 , Figure 1 This is a flowchart of an optimizer quantization method based on a text generation model provided by an embodiment of the present application, including steps S100 to S600. Specifically,

[0067] Step S100: Read the text input tensor of the optimizer, where the text input tensor is floating-point data with the first bit width;

[0068] Step S200: Determine the gradient information of the text input tensor;

[0069] Step S300: performing block processing on the gradient information to obtain a plurality of independent blocks, performing normalization processing on the independent blocks to obtain normalization constants of the independent blocks, and performing quantization processing on the independent blocks according to the normalization constants to obtain quantization results of the independent blocks, wherein the quantization results are integer data of a second bit width, wherein the second bit width is smaller than the first bit width;

[0070] Step S400: performing optimization preprocessing on the quantization result to obtain an optimized quantization result, and using the optimized quantization result as a first optimizer state;

[0071] Step S500: performing inverse quantization processing on the first optimizer state to obtain a second optimizer state, and updating the optimizer according to the second optimizer state;

[0072] Step S600: quantize the second optimizer state to return to the first optimizer state, and store the optimized quantization results of the independent blocks.

[0073] In order to reduce the memory usage of the optimizer in the text generation model and improve the utilization rate of the graphics card, an optimizer quantization method based on the text generation model is proposed in the embodiment of the present application. The optimizer quantization method based on the text generation model includes reading the text input tensor of the optimizer, wherein the text input tensor is floating-point data of the first bit width; then, calculating the gradient information of the text input tensor, and performing block processing on the gradient information to obtain multiple independent blocks; performing normalization processing on each independent block respectively to solve the normalization constant of each independent block. The gradient information is divided into multiple independent blocks, and each independent block can be normalized separately, without the need for synchronous solution of the gradient information of the input text between the central processing unit (CPU) cores. Each independent block is processed in parallel across the CPU cores, thereby improving the quantization accuracy and improving the throughput of the input text received by the optimizer, resulting in faster optimization. The independent block is quantized according to the normalization constant of the independent block to obtain the quantization result of the independent block, wherein the quantization result is integer data of the second bit width. Since each independent block has a corresponding normalization constant, each independent block can be quantized independently. Therefore, the outliers between independent blocks will not affect each other, so that the independent block can isolate the outliers of other different independent blocks, thereby making the quantization process more accurate. In addition, it can also make the outliers more evenly distributed in multiple different independent blocks, with smaller absolute quantization error and relative quantization error for non-uniform distribution.

[0074] In some embodiments of the present application, the text input tensor is 32-bit floating-point data, and the quantization result of the independent block is 8-bit integer data. The quantization result of the independent block is optimized and pre-processed to avoid the data overflow problem caused by the data representation range being narrowed from 32 bits to 8 bits during the quantization process, thereby improving the accuracy of the quantization result of the independent block; the quantization result of the independent block is optimized to obtain an optimized quantization result after optimization processing, and the optimized quantization result is used as the first optimizer state. The first optimizer state is dequantized to obtain a second optimizer state, and the optimizer is updated according to the second optimizer state. The second optimizer state is quantized to return to the first optimizer state, and the optimized quantization result of the independent block is stored. Among them, the first optimizer state is an 8-bit optimizer state, and the second optimizer state is a 32-bit optimizer state.

[0075] By dequantizing the first optimizer state to the second optimizer state and performing an update, the performance level of using the second optimizer state can be maintained, and then the second optimizer state is quantized back to the first optimizer state for storage, which can reduce the graphics memory usage of the optimizer in the text generation model and improve the utilization of the graphics card. The conversion from the first optimizer state to the second optimizer state is performed on the quantization results of the gradient information of each independent block in the register, eliminating the need to slowly copy the quantization results of multiple independent blocks to the graphics card memory or additional temporary memory to perform quantization and dequantization, further reducing the graphics memory usage of the optimizer in the text generation model and achieving higher utilization of graphics card resources.

[0076] In some embodiments of the present application, the gradient information includes the exponential mean value of the gradient and the sum of the squares of the gradient, and the exponential mean value of the gradient and the sum of the squares of the gradient are quantized into 8-bit integers in blocks. Then, the 8-bit optimizer state is dequantized to a 32-bit optimizer state for update, which can maintain the performance level of the 32-bit optimizer state, and then the 32-bit optimizer state is quantized back to an 8-bit optimizer state for storage, which can reduce the video memory usage of the optimizer in the text generation model and improve the utilization rate of the graphics card. This 8-bit to 32-bit conversion is performed element by element on the independent block in the register, without the need for slow copying to the graphics card memory or additional temporary memory to perform quantization and dequantization, further reducing the video memory usage of the optimizer in the text generation model.

[0077] In some embodiments of the present application, in order to solve the problem of data overflow caused by the narrowing of the data representation range of the gradient information of the independent block during the quantization process, the quantization results of the independent block are optimized and preprocessed. By predefining the data structure of the quantization result, the data structure of the quantization result is dynamically quantized. In the present application, the quantization result is integer data of the second bit width, so the data structure of the integer data of the second bit width is defined. The data structure of the integer data of the second bit width includes a sign bit, an exponent bit, an indicator bit, and a linear quantization bit. Specifically, the integer data of the second bit width is defined by the following data structure:

[0078] The sign bit is the first bit of the data structure;

[0079] Exponent bits, used to indicate the size of the exponent bits of integer data by the number of consecutive zero bits;

[0080] Indication bit, set to 1;

[0081] Linear quantization bit, used to indicate the linear quantization value of integer data.

[0082] By defining the integer data structure with the second bit width, the problem of data overflow caused by the narrowing of the data representation range of the gradient information of the independent block during the quantization process can be solved, thereby improving the quantization accuracy. By setting the indicator bit to 1, the data structure below the indicator bit is reserved for the linear quantization bit. By moving the indicator bit, the linear quantization value of the integer data can have 10 -7 exponential or as high as By defining a data structure for integer data of the second bit width, the quantization results of independent blocks are dynamically quantized, allowing both small and large linear quantization values ​​of the integer data to be quantized with high precision, thereby reducing the quantization error generated during the quantization process. Compared with linear quantization, dynamic quantization has better absolute and relative quantization errors for non-uniform distributions. By defining a data structure for integer data of the second bit width, the quantization results of independent blocks are dynamically quantized, enabling optimized preprocessing of the quantization results of the independent blocks, resulting in optimized quantization results.

[0083] In the related art, in the training process of many natural language models and text generation models, the sign bit is usually deleted to increase the range of data variation. In some embodiments of the present application, the data structure of the second-bit-width integer data can be flexibly adjusted according to the optimizer state. In another embodiment, the data structure of the second-bit-width integer data includes a fixed bit of the fraction, an exponent bit, an indicator bit, and a linear quantization bit. Specifically, the second-bit-width integer data is defined by the following data structure:

[0084] The fixed bit of the score is the first bit of the data structure;

[0085] Exponent bits, used to indicate the size of the exponent bits of integer data by the number of consecutive zero bits;

[0086] Indication bit, set to 1;

[0087] Linear quantization bit, used to indicate the linear quantization value of integer data.

[0088] Dynamic quantization is performed on the quantization results of independent blocks by re-adjusting the data structure of the second-bit-width integer data. Since the variance is strictly positive when the Adam optimizer performs the second calculation, the quantization result is also strictly positive. Therefore, the second-bit-width integer data does not require a sign bit. In this embodiment, instead of simply deleting the sign bit, a fixed bit of a fraction is used to replace the sign bit. The fixed bit of the fraction is set as the first bit of the second-bit-width integer data. By extending dynamic quantization through the fixed bit of the fraction, the data variation range of the quantization result can be very effectively covered.

[0089] By defining the integer data structure with the second bit width, the data overflow problem caused by the narrowing of the data representation range of the variance gradient information of the independent block during the quantization process when the Adam optimizer performs the second calculation can be solved, thereby improving the quantization accuracy. By setting the indicator bit to 1, the data structure below the indicator bit is reserved for the linear quantization bit. By moving the indicator bit, the linear quantization value of the integer data can have 10 -7 exponential or as high as By defining the data structure of the integer data of the second bit width, the quantization results of the independent blocks are dynamically quantized, so that both the small and large values ​​of the linear quantization values ​​of the integer data are quantized with high precision, thereby reducing the quantization error generated during the quantization process. Compared with linear quantization, dynamic quantization has better absolute quantization error and relative quantization error for non-uniform distribution. By flexibly adjusting the data structure of the integer data of the second bit width, the quantization results of the variance information of the independent blocks are dynamically quantized when the Adam optimizer performs the second calculation, thereby achieving optimized preprocessing of the quantization results of the independent blocks and obtaining optimized quantization results.

[0090] Reference Figure 2 , Figure 2 This is a schematic diagram of a data structure of integer data of a second bit width provided by an embodiment of the present application. The data structure of integer data of a second bit width includes a sign bit, an exponent bit, an indicator bit, and a linear quantization position. The sign bit is 1, indicating a negative number. The number of zero bits of the exponent bit is 2, so the order of magnitude of the exponent bit is 2. The indicator bit is 1, and the indicator bit 1 indicates that all values ​​after the indicator bit are reserved for the linear quantization bit. Therefore, 1001 are all linear quantization values ​​of integer data represented by the linear quantization bit. By moving the indicator bit, the size of the linear quantization value of the integer data represented by the linear quantization bit can be flexibly changed, so that the linear quantization value of the integer data can have 10 -7 exponential or as high as The accuracy is very effective in covering the range of data variation.

[0091] Reference Figure 3 , Figure 3 This is a flowchart of a method for obtaining a quantization result of an independent block provided by an embodiment of the present application, including steps S310 to S350. Specifically,

[0092] Step S310: converting the text input tensor into a one-dimensional element sequence, and dividing the one-dimensional element sequence into multiple independent blocks of preset interval sizes;

[0093] Step S320: determining a normalization constant for an independent block, and converting the text input tensor into a range of the domain of the target quantized data type according to the normalization constant;

[0094] Step S330: determining a corresponding value of each element of the one-dimensional element sequence in the domain of the target quantized data type;

[0095] Step S340: storing the index corresponding to the corresponding value;

[0096] Step S350: Obtain the quantization result of the independent block according to the index.

[0097] In some embodiments of the present application, the input tensor is divided into blocks to obtain multiple independent blocks, which enables the input tensor to be quantized.

[0098] The gradient information is processed in blocks to obtain multiple independent blocks; each independent block is quantized to obtain the quantization results of each independent block. The quantization process is to convert the gradient information from the floating point data of the first bit width into the integer data of the second bit width. The quantization method saves space by compressing the digital representation, but the quantization cost is the loss of precision. In some embodiments of the present application, the problem of data accuracy in the quantization process is solved by the following method, including: receiving the text input tensor of the optimizer, and converting the text input tensor into a one-dimensional element sequence T, and then dividing the one-dimensional element sequence T into independent blocks of size B, so for a one-dimensional element sequence T with n elements, Independent blocks. Determine one by one Normalization constants for independent blocks are calculated, and the text input tensor is converted to the domain of the target quantization data type according to the normalization constants. The corresponding values ​​of n elements of the one-dimensional element sequence T in the domain of the target quantization data type are determined, the indices corresponding to the corresponding values ​​are stored, and the quantization results of the independent blocks are obtained through the indices.

[0099] In some embodiments of the present application, the normalization constant of an independent block is calculated as follows:

[0100] N b =max(|T b |);

[0101] Where b is the index of an independent block, T is a one-dimensional element sequence, and N b is the normalization constant.

[0102] By calculating the normalization constant of each independent block and independently quantizing the independent block according to the normalization constant of the independent block, since each independent block has a corresponding normalization constant, each independent block can be quantized independently, so the outliers between the independent blocks will not affect each other, so that the independent block can isolate the outliers of other different independent blocks, thereby making the quantization process more accurate. In addition, it can also make the outliers more evenly distributed in multiple different independent blocks, with smaller absolute quantization error and relative quantization error for non-uniform distribution.

[0103] Reference Figure 4 , Figure 4 This is a flowchart of another optimizer quantization method based on a text generation model provided by an embodiment of the present application, including steps S341 to S344. Specifically,

[0104] Step S341: performing denormalization processing on the index;

[0105] Step S342: determining the maximum value of the one-dimensional element sequence;

[0106] Step S343: Determine the quantized output data corresponding to the index by using a binary search method based on the denormalized index and the maximum value of the one-dimensional element sequence;

[0107] Step S344: Determine the quantized output data corresponding to the index determined by the binary method as the quantization result of the independent block.

[0108] In some embodiments of the present application, in the quantization method of an independent block, in order to receive the dequantized one-dimensional element sequence, it is necessary to search for the index and denormalize. Therefore, in this embodiment, after determining the corresponding value of each element of the one-dimensional element sequence in the domain of the target quantization data type and storing the index corresponding to the corresponding value, the quantization method of the independent block also includes performing denormalization on the index, and then determining the maximum value of the one-dimensional element sequence corresponding to each independent block. In the process of performing dynamic quantization on the independent block, it is necessary to divide by the maximum value of the one-dimensional element sequence corresponding to the independent block to normalize the range of the domain of the target quantization data type to [-1, 1]. Based on the denormalized index and the maximum value of the one-dimensional element sequence, the value closest to the index in the domain of the target quantization data type is searched by binary search, and the closest value is used as the quantization output data corresponding to the index. The quantization output data corresponding to the index determined by binary search is determined as the quantization result of the independent block.

[0109] The following is an example to illustrate the method and steps for obtaining the quantization results of independent blocks.

[0110] Quantization saves space by compressing digital representations, but the cost of quantization is loss of precision. Quantization maps k-bit integers to real elements in D dimensions. For example, the IEEE 32-bit floating-point data type will have indexes 0, 1, 2, 3...2 {32}-1 Maps to the domain [-3.4e38,+3.4e38].

[0111] The quantization conversion process is expressed by the following formula:

[0112]

[0113] Among them, Qmap is the target quantized data type, i is the index, and q_i is the corresponding value of the element of the independent block in the domain D.

[0114] To perform general quantization from 32-bit floating point data to 8-bit integer data, it is implemented through steps S1 to S3, specifically:

[0115] Step S1: Calculate the normalization constant N of the independent block b , convert the text input tensor into a one-dimensional element sequence T, and convert the one-dimensional element sequence T into the target quantized data type Q map The range of domain D,

[0116] Step S2: Determine the corresponding value q_i of each element of the one-dimensional element sequence in the domain D;

[0117] Step S3: Store the index corresponding to the corresponding value, search the index and perform denormalization on the index, determine the quantized output data according to the denormalized index, and store the quantized output data T of the index i corresponding to q_i D .

[0118] Quantized output data T D It is obtained by the following formula:

[0119]

[0120] To perform the dynamic quantization process, first divide by the absolute maximum value N=max(|T b |) to normalize the dynamic quantization range [-1, 1], where the absolute maximum is the maximum value among all independent blocks. By dividing by the absolute maximum, the maximum optimizer state is guaranteed, while producing low quantization error for both small and large values.

[0121] Then, the closest value is found through binary search, and the value is determined as the quantized output data corresponding to the index, and the quantized output data corresponding to the index determined through binary search is determined as the quantization result of the independent block.

[0122] The quantization output data corresponding to the index is determined by binary division to determine the quantization result of the independent block through the following formula:

[0123]

[0124] By calculating the normalization constant of each independent block and independently quantizing the independent block according to the normalization constant of the independent block, since each independent block has a corresponding normalization constant, each independent block can be quantized independently, so that the outliers between the independent blocks will not affect each other. Therefore, the independent block can isolate the outliers of other different independent blocks, making the quantization process more accurate. In addition, it can also make the outliers more evenly distributed in multiple different independent blocks, with smaller absolute quantization error and relative quantization error for non-uniform distribution.

[0125] Reference Figure 5 , Figure 5 This is a flowchart of another optimizer quantization method based on a text generation model provided by an embodiment of the present application, including steps S800 to S840. Specifically,

[0126] Step S800: Input the text input tensor into the embedding layer module to obtain an embedding vector;

[0127] Step S810: Input the embedding vector into the Transformer model and perform a forward operation to obtain the optimizer loss function;

[0128] Step S820: performing back propagation calculation on the loss function to optimize the loss function value;

[0129] Step S830: Determine the gradient of the optimizer, and obtain the gradient of the loss function for the text generation model based on the loss function and the gradient;

[0130] Step S840: quantize the gradient.

[0131] In some embodiments of the present application, in order to solve the stability problem of the text generation model training process, the application layer is normalized. Therefore, after reading the input tensor of the optimizer, the text input tensor is input into the embedding layer of the optimizer, and each element corresponding to the text input tensor outputs the corresponding embedding vector, which is input into the Transformer model, and a forward operation is performed to obtain the loss function of the optimizer. During the text generation model training process, the loss function value is usually used to monitor the training progress of the model. When the loss function value is small enough or tends to be stable, the model is considered to have converged. By performing backpropagation calculation on the loss function, the loss function value is gradually reduced to optimize the loss function value. By optimizing the loss function value, the text generation model has a better fitting effect on the data of the text input tensor. The gradient of the optimizer is determined, and the gradient of the loss function for the text generation model is obtained based on the loss function and the gradient; and the gradient is quantized.

[0132] After reading the input tensor of the optimizer, the text input tensor is input into the embedding layer of the optimizer for the above preprocessing, so that the text generation model maintains a variance of approximately 1 during both initialization and training, thereby reducing the maximum gradient size and improving the quantization accuracy of the text generation model and the stability and reliability of the data.

[0133] Reference Figure 6 , Figure 6 This is a flowchart of another optimizer quantization method based on a text generation model provided by an embodiment of the present application, including steps S900 to S950. Specifically,

[0134] Step S900: normalizing the input text tensor with highly uneven distribution;

[0135] Step S910: Input the normalized text input tensor into the embedding layer module to obtain an embedding vector;

[0136] Step S920: Input the embedding vector into the Transformer model and perform a forward operation to obtain the optimizer's loss function;

[0137] Step S930: performing back propagation calculation on the loss function to optimize the loss function value;

[0138] Step S940: Determine the gradient of the optimizer, and obtain the gradient of the loss function for the text generation model based on the loss function and the gradient;

[0139] Step S950: quantize the gradient.

[0140] In some embodiments of the present application, in order to avoid the existence of highly uneven distribution of input tensors, the highly uneven distribution of inputs is normalized to improve the stability of the text generation model optimization process. Therefore, in the present application, the optimizer quantization method based on the text generation model also includes: before the normalized text input tensor is input into the embedding layer module, the input highly unevenly distributed text input tensor is first normalized to support more aggressive quantization, thereby avoiding extreme gradient changes. The normalized text input tensor is input into the embedding layer module to obtain an embedding vector.

[0141] Using Xavier uniform initialization to initialize the stable embedding layer and applying layer normalization before adding positional embeddings ensures that the text generation model maintains a variance of approximately 1 during both initialization and training, thereby reducing the maximum gradient magnitude and improving the quantization accuracy of the text generation model and the stability and reliability of the data. In addition, normalizing the highly uneven distribution of the input ensures that the uniform distribution of the input tensor has fewer extreme values ​​than a normal distribution, further reducing the maximum gradient magnitude.

[0142] In some embodiments of the present application, determining the gradient of the optimizer and obtaining the gradient of the loss function for the text generation model based on the loss function and the gradient includes:

[0143] Let the loss function be L and the text generation model parameters be W, where W includes all weights and biases. To calculate the gradient of the loss function with respect to the text generation model parameters W, we can use the chain rule to perform the derivation. Specifically, for each parameter W i , the gradient can be calculated by the following steps:

[0144] First, the loss function value L is calculated based on the predicted output tensor and the actual output tensor label of the text generation model;

[0145] Find the gradient of the loss function L with respect to the predicted output tensor. Specifically, calculate the gradient of the loss function L with respect to the predicted output tensor of the text generation model, commonly called the error term. The error term represents the slope of the loss function L at the current predicted output tensor and is the starting point of the chain rule.

[0146] Find the gradient of the internal variables of the text generation model with respect to the text generation model parameters W, specifically through matrix multiplication and element-by-element multiplication.

[0147] Multiply the gradients obtained in the above steps to get the loss function L for the text generation model parameter W i gradient.

[0148] Reference Figure 7 , Figure 7 The present invention provides a structural diagram of an optimizer quantization device based on a text generation model. The optimizer quantization device based on a text generation model includes a reading module, a gradient calculation module, a dynamic quantization module, an embedding layer module, and an inverse quantization module.

[0149] The reading module is used to read the text input tensor of the optimizer, where the text input tensor is floating-point data with the first bit width;

[0150] Gradient calculation module, used to determine the gradient information of the text input tensor;

[0151] The block-by-block quantization module is used to process the gradient information in blocks to obtain multiple independent blocks, and is also used to normalize the independent blocks to obtain normalization constants of the independent blocks, and quantize the independent blocks according to the normalization constants to obtain quantization results of the independent blocks, wherein the quantization results are integer data of the second bit width.

[0152] A dynamic quantization module is used to perform optimization preprocessing on the quantization result to obtain an optimized quantization result, and use the optimized quantization result as the first optimizer;

[0153] The dequantization module is used to dequantize the first optimizer state to obtain the second optimizer state, and update the second optimizer state to the optimizer state; quantize the second optimizer state to return to the first optimizer state, and store the quantization results of the independent blocks.

[0154] The block-by-block quantization module divides the gradient information into multiple independent blocks and can perform normalization processing on each independent block separately. There is no need for synchronous solution of the gradient information of the input text between CPU cores. Each independent block is processed in parallel across CPU cores, which improves the quantization accuracy. The independent block is quantized according to the normalization constant of the independent block to obtain the quantization result of the independent block. Since each independent block has a corresponding normalization constant, each independent block can be quantized independently. Therefore, the outliers between independent blocks will not affect each other. Therefore, the independent block can isolate the outliers of other different independent blocks, making the quantization process more accurate. In addition, it can also make the outliers more evenly distributed across multiple different independent blocks, with smaller absolute quantization error and relative quantization error for non-uniform distribution.

[0155] The dynamic quantization module optimizes the quantization results, effectively preventing data overflow caused by a narrowing of the data representation range of gradient information for independent blocks during the quantization process, thereby improving quantization accuracy. By defining the data structure of the quantization results, both the small and large linear quantization values ​​of the integer data are quantized with high precision, thereby reducing the quantization error generated during the quantization process. Compared with linear quantization, dynamic quantization has better absolute and relative quantization errors for non-uniform distributions. By defining the data structure of integer data with a second bit width, the quantization results of independent blocks are dynamically quantized, thereby achieving optimized preprocessing of the quantization results of the independent blocks and obtaining optimized quantization results.

[0156] The dequantization module maintains the performance level of using the second optimizer state by dequantizing the first optimizer state to the second optimizer state and performing an update. The second optimizer state is then quantized back to the first optimizer state for storage. This reduces the video memory usage of the optimizer in the text generation model and improves graphics card utilization. The conversion from the first optimizer state to the second optimizer state is performed on the quantization results of the gradient information of each independent block in the register, eliminating the need to slowly copy the quantization results of multiple independent blocks to the graphics card memory or additional temporary memory to perform quantization and dequantization. This further reduces the video memory usage of the optimizer in the text generation model and improves graphics card resource utilization.

[0157] In order to avoid the existence of highly uneven distribution of input tensors, the highly uneven distribution of inputs is normalized to improve the stability of the text generation model optimization process. In an embodiment of the present application, an optimizer quantization device based on a text generation model is provided, which also includes an embedding layer module. After reading the input tensor of the optimizer, the text input tensor is input into the embedding layer of the optimizer quantization device. The embedding layer is used to normalize the input highly unevenly distributed text input tensor to support more aggressive quantization, thereby avoiding extreme gradient changes. The normalized text input tensor is input into the embedding layer module to obtain an embedding vector. The stable embedding layer is initialized using Xavier uniform initialization, and layer normalization is applied before adding position embedding, so that the text generation model maintains a variance of approximately 1 during initialization and training, thereby reducing the maximum gradient size and improving the quantization accuracy of the text generation model and the stability and reliability of the data.

[0158] In some embodiments of the present application, in an optimizer quantization device based on a text generation model, a reading module receives a text input tensor of 32-bit floating point data type, inputs the text input tensor into an embedding layer, and normalizes the highly unevenly distributed text input tensor to support more aggressive quantization, thereby avoiding extreme gradient changes. The gradient calculation module calculates the exponential average and the square sum of the gradient of the text input tensor, and the block-by-block quantization module quantizes this information into 8-bit integer data in blocks; then, the dequantization module dequantizes the 8-bit optimizer state into a 32-bit optimizer state, performs an update, and can maintain the performance level of using the second optimizer state. The optimizer state is then quantized back to 8 bits for storage, which can reduce the graphics card memory usage of the optimizer in the text generation model and improve the utilization of the graphics card. The conversion of the 8-bit optimizer state to the 32-bit optimizer is performed on the quantization results of the gradient information of each independent block in the register, without the need to slowly copy the quantization results of multiple independent blocks to the graphics card memory or additional temporary memory to perform quantization and dequantization, further reducing the graphics card memory usage of the optimizer in the text generation model and making the graphics card resource utilization higher.

[0159] Reference Figure 8 , Figure 8This is a structural diagram of a controller 1000 provided in an embodiment of the present application, including a processor 1001, which can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the optimizer quantization method based on the text generation model provided in an embodiment of the present application; a memory 1002, which can be implemented in the form of a read-only memory 1002 (Read Only Memory, ROM), a static storage device, a dynamic storage device or a random access memory 1002 (Random Access Memory, RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the embodiments of the present application; the input / output interface 1003 is used to implement information input and output; the communication interface 1004 is used to implement communication interaction between this device and other devices, and communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); a bus transmits information between the various components of the device (such as the processor 1001, memory 1002, input / output interface 1003 and communication interface 1004); wherein the processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device through the bus.

[0160] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer-readable storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer-readable storage medium is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0161] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. An optimizer quantization method based on a text generation model, characterized in that: include: Read the text input tensor of the optimizer, where the text input tensor is floating-point data with the first bit width; Determining gradient information of the text input tensor; performing block processing on the gradient information to obtain a plurality of independent blocks, performing normalization processing on the independent blocks to obtain normalization constants of the independent blocks, and performing quantization processing on the independent blocks according to the normalization constant to obtain quantization results of the independent blocks, wherein the quantization results are integer data of a second bit width, wherein the second bit width is smaller than the first bit width; Performing optimization preprocessing on the quantization result to obtain an optimized quantization result, and using the optimized quantization result as a first optimizer state; performing dequantization processing on the first optimizer state to obtain a second optimizer state, and updating the optimizer according to the second optimizer state; quantizing the second optimizer state to return to the first optimizer state, and storing the optimized quantization result of the independent block; Obtaining the quantization result of the independent block includes: Converting the text input tensor into a one-dimensional element sequence, and dividing the one-dimensional element sequence into a plurality of independent blocks of a preset interval size; Determining a normalization constant for the independent block, and converting the text input tensor to a range of a domain of a target quantized data type according to the normalization constant; determining a corresponding value of each element of the one-dimensional sequence of elements in the domain of the target quantized data type; Storing the index corresponding to the corresponding value; Obtaining a quantization result of the independent block according to the index; After storing the index corresponding to the corresponding value, the method further includes: performing denormalization on the index; determining a maximum value of the one-dimensional sequence of elements; Determining quantized output data corresponding to the index by a binary search method according to the denormalized index and the maximum value of the one-dimensional element sequence; Determining the quantized output data corresponding to the index determined by the binary method as the quantization result of the independent block; The optimizer quantization method based on the text generation model also includes: Input the text input tensor into the embedding layer module to obtain an embedding vector; Input the embedding vector into the Transformer model and perform a forward operation to obtain the loss function of the optimizer; Performing backpropagation calculation on the loss function to optimize the loss function value; Determining the gradient of the optimizer, and obtaining the gradient of the loss function for the text generation model based on the loss function and the gradient; The gradient is quantized.

2. The optimizer quantization method based on the text generation model according to claim 1, characterized in that The second bit-width integer data is defined by the following data structure: A sign bit, which is the first bit of the data structure; exponent bits, used to indicate the size of the exponent bits of the integer data by a number of consecutive zero bits; Indication bit, set to 1; The linear quantization bit is used to indicate the linear quantization value of the integer data.

3. The optimizer quantization method based on the text generation model according to claim 1, characterized in that: The second bit-width integer data is defined by the following data structure: The fixed bit of the fraction is the first bit of the data structure; exponent bits, used to indicate the size of the exponent bits of the integer data by a number of consecutive zero bits; Indication bit, set to 1; The linear quantization bit is used to indicate the linear quantization value of the integer data.

4. The optimizer quantization method based on the text generation model according to claim 1, characterized in that Before inputting the text input tensor into the embedding layer module, the method further includes: Normalize the input text input tensor with highly uneven distribution.

5. An optimizer quantization device based on a text generation model, characterized in that: include: A reading module, configured to read a text input tensor of an optimizer, wherein the text input tensor is floating-point data of the first bit width; A gradient calculation module, configured to determine gradient information of the text input tensor; a block-by-block quantization module, configured to perform block processing on the gradient information to obtain a plurality of independent blocks, and further configured to perform normalization processing on the independent blocks to obtain normalization constants of the independent blocks, and perform quantization processing on the independent blocks according to the normalization constants to obtain quantization results of the independent blocks, wherein the quantization results are integer data of a second bit width, wherein the second bit width is smaller than the first bit width; A dynamic quantization module is used to perform optimization preprocessing on the quantization result to obtain an optimized quantization result; An embedding layer module, configured to preprocess the optimization quantization result to obtain a first optimizer state; a dequantization module, configured to perform dequantization processing on the first optimizer state to obtain a second optimizer state, and update the second optimizer state to the optimizer state; perform quantization processing on the second optimizer state to return to the first optimizer state, and store the quantization result of the independent block; Obtaining the quantization result of the independent block includes: Converting the text input tensor into a one-dimensional element sequence, and dividing the one-dimensional element sequence into a plurality of independent blocks of a preset interval size; Determining a normalization constant for the independent block, and converting the text input tensor to a range of a domain of a target quantized data type according to the normalization constant; determining a corresponding value of each element of the one-dimensional sequence of elements in the domain of the target quantized data type; Storing the index corresponding to the corresponding value; Obtaining a quantization result of the independent block according to the index; After storing the index corresponding to the corresponding value, the method further includes: performing denormalization on the index; determining a maximum value of the one-dimensional sequence of elements; Determining quantized output data corresponding to the index by a binary search method according to the denormalized index and the maximum value of the one-dimensional element sequence; Determining the quantized output data corresponding to the index determined by the binary method as the quantization result of the independent block; The optimizer quantization method based on the text generation model also includes: Input the text input tensor into the embedding layer module to obtain an embedding vector; Input the embedding vector into the Transformer model and perform a forward operation to obtain the loss function of the optimizer; Performing backpropagation calculation on the loss function to optimize the loss function value; Determining the gradient of the optimizer, and obtaining the gradient of the loss function for the text generation model based on the loss function and the gradient; The gradient is quantized.

6. A controller, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the optimizer quantization method based on the text generation model according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: Computer-executable instructions are stored, and the computer-executable instructions are used to execute the optimizer quantization method based on the text generation model according to any one of claims 1 to 4.

Images