Text generation method and device based on large model quantization, equipment and storage medium

Abstract

The embodiment of the application provides a text generation method and device based on large model quantization, equipment and storage medium, which are applied to the Ascend computing architecture. The original text is input into the target large language model, the target large language model includes a plurality of feature extraction layers which are sequentially cascaded, each feature extraction layer includes a full connection layer, through the plurality of feature extraction layers which are sequentially cascaded in the target large language model, the text features corresponding to the original text are subjected to layer-by-layer incremental feature extraction, and the weight parameters and activation values of the full connection neurons in each feature extraction layer are subjected to quantization processing in cooperation with the quantization method of the Ascend computing architecture, so as to output target text features, simplify the calculation in the full connection stage, and thus realize lightweight calculation under the large language model. The target text is generated based on the target text features, the lightweight large language model is used for the text generation task, the complex calculation of the large model in the text reasoning process is reduced, and the efficiency of the text generation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text generation method and device based on large model quantization, equipment and a storage medium. BACKGROUND

[0002] Under the background of big data, the structure and calculation of large language models are increasingly complex, and the amount of calculation grows exponentially. Although neural network accelerators have significant advantages in terms of calculation acceleration, parallel architecture, and model training, the computing power resources of neural network accelerators are limited, so it is necessary to accelerate the model training and inference process and reduce the dependence on storage resources on the basis of limited computing power resources.

[0003] In related technologies, large language models are used for text generation, such as generating answers to question texts and summaries of long texts. However, the model parameter scale of large language models is large and the model calculation is complex, resulting in low efficiency of text generation. Therefore, how to improve the efficiency of text generation has become a problem to be solved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a text generation method and device based on large model quantization, equipment and a storage medium, which aims to improve the efficiency of text generation.

[0005] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application provides a text generation method based on large model quantization applied to the Ascend computing architecture, the method comprising:

[0006] obtaining an original text;

[0007] inputting the original text into a target large language model, the target large language model comprising a plurality of feature extraction layers cascaded in sequence, each feature extraction layer comprising a fully connected layer;

[0008] performing layer-by-layer incremental feature extraction on the text features corresponding to the original text through the plurality of feature extraction layers cascaded in sequence in the target large language model, and determining the candidate text features output by the feature extraction layer of the last layer as target text features;

[0009] wherein the feature extraction layer of each layer is configured to perform attention calculation on the input text features to obtain attention text features, perform activation processing on the attention text features to obtain text activation values, perform quantization processing on the text activation values to obtain target activation values, and output the candidate text features of the current layer according to the target activation values and the target weights obtained by quantizing the weight parameters of the corresponding fully connected layer;

[0010] The process of the feature extraction of the layer-by-layer increment is to take the candidate text features extracted in the last layer as the feature base data of the next layer for feature extraction.

[0011] The target text feature is generated based on the target text.

[0012] In some embodiments, the target weight is calculated according to the following steps:

[0013] The weight parameters of the full connection layer are calculated by mean value to obtain a weight mean value;

[0014] The weight parameters of the full connection layer are calculated by variance based on the weight mean value to obtain a weight variance value;

[0015] The weight quantization parameters are determined based on the weight mean value and the weight variance value;

[0016] The weight parameters of the full connection layer are quantized based on the weight quantization parameters to obtain the target weight.

[0017] In some embodiments, the quantization of the weight parameters of the full connection layer based on the weight quantization parameters to obtain the target weight comprises:

[0018] The weight parameters of the full connection layer are initially quantized based on the weight quantization parameters to obtain an initial weight;

[0019] The initial weight is rounded to obtain a first intermediate weight;

[0020] The first intermediate weight is weight-clipped to obtain a second intermediate weight;

[0021] The second intermediate weight is re-quantized based on the weight quantization parameters to obtain the target weight.

[0022] In some embodiments, the determination of the weight quantization parameters based on the weight mean value and the weight variance value comprises:

[0023] The weight mean square deviation is obtained based on the weight variance value;

[0024] The weight mean value and the weight mean square deviation are calculated by mean value to obtain the weight quantization parameters.

[0025] In some embodiments, the quantization of the text activation value to obtain the target activation value comprises:

[0026] The text activation value and a preset constant are screened to obtain an activation quantization parameter;

[0027] Quantize the text activation value according to the activation quantization parameter to obtain the target activation value.

[0028] In some embodiments, the quantizing the text activation value according to the activation quantization parameter to obtain the target activation value comprises:

[0029] Preliminary quantizing the text activation value according to the activation quantization parameter to obtain an initial activation value;

[0030] Integrating the initial activation value to obtain a first intermediate activation value;

[0031] Activation clipping the first intermediate activation value to obtain a second intermediate activation value;

[0032] Requantizing the second intermediate activation value according to the activation quantization parameter to obtain the target activation value.

[0033] In some embodiments, the screening the text activation value and a preset constant to obtain an activation quantization parameter comprises:

[0034] Selecting the maximum text activation value as a candidate activation value;

[0035] Comparing the candidate activation value with the preset constant;

[0036] If the candidate activation value is greater than or equal to the preset constant, taking the candidate activation value as the activation quantization parameter;

[0037] If the candidate activation value is less than the preset constant, taking the preset constant as the activation quantization parameter.

[0038] To achieve the above-mentioned purpose, a second aspect of the embodiment of the present application proposes a text generation device based on large model quantization, applied to the Ascend computing architecture, the device comprises:

[0039] An acquisition module is configured to acquire an original text.

[0040] An input module is configured to input the original text into a target large language model, wherein the target large language model comprises a plurality of feature extraction layers that are sequentially cascaded, and each feature extraction layer comprises a full connection layer.

[0041] A feature extraction module is configured to perform layer-by-layer incremental feature extraction on text features corresponding to the original text based on the plurality of feature extraction layers that are sequentially cascaded in the target large language model, and determine candidate text features output by the last layer of the feature extraction layer as target text features.

[0042] The feature extraction layer of each level is configured to perform attention calculation on the input text feature to obtain an attention text feature, perform activation processing on the attention text feature to obtain a text activation value, perform quantization processing on the text activation value to obtain a target activation value, and output a candidate text feature of the current level according to the target activation value and a target weight obtained by quantizing a weight parameter of the full connection layer corresponding to the target activation value.

[0043] The process of the layer-by-layer incremental feature extraction is to take the candidate text feature extracted by a previous layer as feature base data for feature extraction of a next layer.

[0044] The text generation module is configured to generate a target text based on the target text feature.

[0045] To achieve the above object, a third aspect of embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0046] To achieve the above object, a fourth aspect of embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0047] The text generation method based on large model quantization, the text generation device based on large model quantization, the electronic equipment and the computer readable storage medium provided by the embodiments of the present application adopt the Ascend computing architecture as a neural network accelerator to provide high-performance computing support for artificial intelligence applications. The original text is obtained to perform a text generation task on the Ascend computing architecture using the original text to accelerate the inference of the text generation task. The original text is input into a target large language model, and through a plurality of feature extraction layers in the target large language model that are sequentially cascaded, the text features corresponding to the original text are extracted layer by layer in an incremental manner, different levels of text features can be extracted, the context semantics of the original text is fully understood, the expression ability of the large language model is improved, and the candidate text features output by the last level of feature extraction layer are determined as target text features to capture complex context relationships in the original text, thereby generating a coherent target text. The model parameter scale of the large language model is large, the model calculation is complex, and the computing power resources of the neural network accelerator are limited. The weight quantization operation and the activation quantization operation adapted to the Ascend computing architecture reduce the model parameter scale in the embodiments of the present application, simplify the complex calculation of the model on the text, reduce the consumption of storage resources and computing power resources of the Ascend computing architecture, and thus improve the efficiency of text generation. The feature extraction layer of each level is used to perform attention calculation on the input text features to focus on key text features in the original text that play an important role in text generation, obtain attention text features, and perform activation processing on the attention text features to enable the model to learn complex nonlinear dependency relationships in the text, obtain text activation values, perform quantization processing on the text activation values to reduce the scale of the text activation values, obtain target activation values, and output the candidate text features of the current level according to the target activation values and the target weights obtained by quantizing the weight parameters of the corresponding fully connected layer. By quantizing the weight parameters of the fully connected layer, the model parameter scale of the large language model can be reduced, and the complexity of weight calculation can be reduced to obtain target weights. According to the target activation values and the target weights, the candidate text features of the current level are output, and the complex calculation of the fully connected layer on the text features is simplified. The target text is generated based on the target text features, the complex calculation operation of the fully connected stage is simplified by quantizing the fully connected layers of each feature extraction layer, and the model parameter scale of the large language model is reduced, thereby realizing lightweight calculation under the large language model. By using the lightweight large language model to perform the text generation task, the complex calculation of the large model in the text inference process is reduced, the efficiency of text generation is improved, and the consumption of storage resources and computing power resources of the Ascend computing architecture is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of the text generation method based on large model quantization provided by the embodiments of the present application;

[0049] Figure 2 is a flowchart of step S130 in Figure 1

[0050] Figure 3 is a flowchart of step S230 in Figure 2

[0051] Figure 4 is a flowchart of step S240 in Figure 2

[0052] Figure 5 is another flowchart of step S130 in Figure 1

[0053] Figure 6 is a flowchart of step S510 in Figure 5

[0054] Figure 7 is a flowchart of step S520 in Figure 5

[0055] Figure 8 is a structural schematic diagram of a text generation apparatus based on large model quantization provided by an embodiment of the present application.

[0056] Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0058] It should be noted that although the functional modules are divided in the apparatus schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the apparatus or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0060] ​​​​​​Under the background of big data, the structure and calculation of large language models become more and more complex, and the amount of calculation increases exponentially. Although neural network accelerators have significant advantages in calculation acceleration, parallel architecture, and model training, the computing power resources of neural network accelerators are limited, and therefore it is necessary to accelerate the model training and inference process and reduce the dependence on storage resources on the basis of limited computing power resources.

[0061] In related technologies, large language models are used for text generation, such as generating answers to question texts and summaries of long texts by using large language models. However, the model parameter scale of large language models is large, and the model calculation is complex, which makes the efficiency of text generation low. Therefore, how to improve the efficiency of text generation has become a problem to be solved.

[0062] Based on this, the embodiment of the present application provides a text generation method based on large model quantization, a text generation device based on large model quantization, an electronic device, and a computer readable storage medium, aiming to improve the efficiency of text generation.

[0063] The text generation method based on large model quantization, the text generation device based on large model quantization, the electronic device, and the computer readable storage medium provided by the embodiment of the present application are specifically explained by the following embodiments. First, the text generation method based on large model quantization in the embodiment of the present application is described.

[0064] The text generation method based on large model quantization provided by the embodiment of the present application relates to the technical field of artificial intelligence. The text generation method based on large model quantization provided by the embodiment of the present application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; and the software can be an application for implementing the text generation method based on large model quantization, etc., but is not limited to the above forms.

[0065] The application is operable with numerous general purpose or special purpose computer system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0066] Figure 1 is an optional flowchart of the text generation method based on large model quantization provided by the embodiment of the application. The text generation method based on large model quantization is applied to the Ascend computing architecture, which is an artificial intelligence computing acceleration platform. The Ascend computing architecture has high computing power and can provide high-performance computing support for artificial intelligence applications. The method can include but is not limited to steps S110 to S140.

[0067] Step S110, obtaining an original text;

[0068] Step S120, inputting the original text into a target large language model, the target large language model including a plurality of feature extraction layers cascaded in sequence, each feature extraction layer including a full connection layer;

[0069] Step S130, performing layer-by-layer incremental feature extraction on the text features corresponding to the original text through the plurality of feature extraction layers cascaded in sequence in the target large language model, and determining the candidate text features output by the last layer of feature extraction layer as target text features;

[0070] Wherein, the feature extraction layer of each layer is used to perform attention calculation on the input text features to obtain attention text features, and perform activation processing on the attention text features to obtain text activation values, perform quantization processing on the text activation values to obtain target activation values, and output the candidate text features of the current layer according to the target activation values and the target weights obtained by quantizing the weight parameters of the corresponding full connection layer;

[0071] Wherein, the process of layer-by-layer incremental feature extraction is to take the candidate text features extracted by the previous layer as the feature basis data for feature extraction of the next layer;

[0072] At step S140, the target text is generated based on the target text features.

[0073] At step S110 of some embodiments, based on a text generation task, text data is obtained from different data sources such as the Internet, social media, news, research literature, etc., to obtain original text, so as to automatically create or generate new target text based on the original text. The text generation task can be a dialogue generation task, a text summary generation task, machine translation, etc. In the dialogue generation task, the original text can be a question text, and the target text can be an answer text matching the question text; in the text summary generation task, the original text can be a long text, and the target text can be a summary text of the long text; in the machine translation task, the original text can be a source language text, and the target text can be a target language text.

[0074] At step S120 of some embodiments, the original text is input to the target large language model for text generation. Based on the Ascend computing architecture, an Ascend 910A computing cluster is built, and based on the Ascend 910A computing cluster, Python 3.9.18, Mindspore 2.3.0rc2, and CANN 8.0rc1 development software are installed to build a simulation platform. According to the operators provided by MindSpore, the target large language model is constructed. The target large language model includes an input layer, a decoding layer, and an output layer. The input layer is used for tokenization processing of the original text to obtain a text field, assigns a unique integer number to the text field according to a preset vocabulary table to obtain a field index, performs word embedding processing on the field index to convert the field index into a vector, achieves feature dispersion effect, and captures semantic relationships between text fields to obtain text features of the original text. The weight parameters of the input layer will participate in the training and inference process to better adapt to the context semantics in the large language model. The decoding layer is a key network layer of the large language model, which is used to further extract deep semantic information contained in the text features and fully understand the context semantics of the original text. The decoding layer includes a plurality of feature extraction layers that are cascaded in sequence, each feature extraction layer adopts a transformer structure, and the number of layers of the feature extraction layer configured according to the text generation task is repeated for multiple layers of calculation. The number of layers of the feature extraction layer can be configured by itself according to actual conditions, such as 32. Each feature extraction layer includes an attention layer and a multi-layer perceptron, the attention layer includes a multi-head attention mechanism and a feedforward network, and at least one of the attention layer and the multi-layer perceptron includes one or more fully connected layers. For example, the multi-layer perceptron includes 2 consecutive fully connected layers. The output layer is an output linear normalization layer, which is used to normalize the output result to the same dimension as the preset vocabulary table to obtain the probability distribution of each output field.

[0075] In step S130 of some embodiments, the text features corresponding to the original text are incrementally extracted layer by layer through a plurality of feature extraction layers in the target large language model, the candidate text features extracted in the previous layer are taken as the feature basis data for feature extraction in the next layer, so as to gradually extract and refine the text features and enhance the understanding ability of the large language model to the original text. The candidate text features output by the last feature extraction layer not only can fully capture the complex context semantic features in the original text, but also can reduce the computational complexity of the subsequent steps. In order to improve the accuracy and efficiency of text generation, the candidate text features output by the last feature extraction layer are determined as the target text features.

[0076] Specifically, the text features corresponding to the original text are extracted through the first feature extraction layer to obtain the candidate text features output by the first feature extraction layer. The candidate text features output by the previous feature extraction layer are taken as the feature basis data for feature extraction in the current feature extraction layer to obtain the candidate text features output by the current feature extraction layer. The above steps are repeated until the candidate text features output by the last feature extraction layer are obtained, which are taken as the target text features.

[0077] The large language model has a large parameter size and a complex calculation, which makes the efficiency of model training and inference low. If text generation is performed through the large language model, the efficiency of text generation will be low. In order to improve the efficiency of text generation, the large language model needs to be lightened to improve the training efficiency and inference efficiency of the large language model to text data. In the embodiments of the present application, the full connection layer is quantized to lighten the large language model with a large parameter and complex calculation, accelerate the model training and inference process on the basis of limited computing resources, and reduce the consumption of storage resources and computing resources in the training and inference process, thereby improving the model execution efficiency and further improving the text generation efficiency.

[0078] In the embodiments of the present application, the quantization of the full connection layer includes weight quantization and activation quantization. The weight quantization of the weight parameters of the full connection layer can reduce the model parameter size of the large language model, thereby simplifying the calculation related to the weight parameters. The activation quantization of the activation values of the full connection layer can reduce the number size of the activation values, simplify the calculation associated with the activation values, and further simplify the complex calculation involved in the full connection layer, thereby improving the efficiency of model training and inference.

[0079] Specifically, the input text features are subjected to attention calculation by the multi-head attention mechanism of each level of the feature extraction layer to focus on the text features that play a key role in the text generation task, and attention text features are obtained. The full connection layer contains an activation function, which can be a ReLU activation function, a sigmoid activation function, a softmax function, etc. The activation function is used to activate the attention text features to obtain text activation values, and the text activation values are quantized to obtain target activation values. The weight parameters of the full connection layer of the corresponding level are quantized to obtain target weights. The quantized full connection calculation is converted into matrix multiplication operation of the target activation values and the target weights. The result of the matrix multiplication operation is the output of the full connection layer. If the full connection layer is the last layer of the feature extraction layer of the current level, the result of the matrix multiplication operation is taken as the candidate text features of the current level. If the full connection layer is not the last layer of the feature extraction layer of the current level, the result of the matrix multiplication operation is subjected to feature extraction until the output of the last layer of the feature extraction layer of the current level is obtained, and the output is taken as the candidate text features of the current level.

[0080] In some embodiments, each feature extraction layer includes a multi-head self-attention mechanism, a feedforward network, a first full connection layer, and a second full connection layer. There is a first residual connection between the input of the multi-head self-attention mechanism and the output of the feedforward network. There is a second residual connection between the input of the first full connection layer and the output of the second full connection layer. The feature extraction layer of each level is used to perform attention calculation on the input text features based on the multi-head self-attention mechanism to obtain first attention features, perform feedforward calculation on the first attention features based on the feedforward network to obtain second attention features, perform feature fusion on the first attention features and the second attention features based on the first residual connection to obtain attention text features, perform activation processing on the attention text features based on the first full connection layer to obtain first text activation values of the first full connection layer, quantize the first text activation values to obtain first target activation values, determine a first output of the first full connection layer according to the first target activation values and first target weights obtained by quantizing the weight parameters of the first full connection layer, perform activation processing on the first output based on the second full connection layer to obtain second text activation values, quantize the second text activation values to obtain second target activation values, determine a second output of the second full connection layer according to the second target activation values and second target weights obtained by quantizing the weight parameters of the second full connection layer, and fuse the first output and the second output based on the second residual connection to output the candidate text features of the current level.

[0081] The quantized full connection calculation is represented as:

[0082]

[0083] wherein Y is the output of the fully connected layer; m and n are the matrix length and the matrix width of the target weight respectively, i and j are the i-th row and the j-th column of the target weight respectively; is the element of the i-th row and the j-th column of the target weight; is the j-th column vector of the target activation value.

[0084] Referring to Figure 2 In some embodiments, the calculation process of the target activation value in step S130 can include, but is not limited to, steps S210 to S240:

[0085] In step S210, the mean value of the weight parameters of the fully connected layer is calculated to obtain a weight mean value;

[0086] In step S220, the variance of the weight parameters of the fully connected layer is calculated according to the weight mean value to obtain a weight variance value;

[0087] In step S230, the weight quantization parameter is determined according to the weight mean value and the weight variance value;

[0088] In step S240, the weight parameters of the fully connected layer are quantized according to the weight quantization parameter to obtain the target weight.

[0089] In step S210 of some embodiments, the number of input channels and the number of output channels of the fully connected layer are obtained, and the number of parameters of the weight parameters of the fully connected layer is determined according to the number of input channels and the number of output channels. If the number of input channels is p and the number of output channels is q, the number of parameters of the weight parameters is q*p. In order to reduce the size of the number of weight parameters, for each fully connected layer in a large language model, the weight mean value of each fully connected layer is obtained by calculating the mean value of the weight parameters of the fully connected layer according to the number of parameters of the weight parameters of the fully connected layer, so that the weight mean value can represent the weight parameters of the entire fully connected layer. The weight mean value of the k-th fully connected layer is represented as:

[0090]

[0091] wherein mean represents the mean value calculation; W k is the weight parameter of the k-th fully connected layer; is the weight mean value of the k-th fully connected layer.

[0092] For example, the fully connected layer includes three weight parameters, which are w1j, w2j and w3j respectively, and the number of parameters is 3, then the weight mean value of the fully connected layer is represented as: (w1j+w2j+w3j) / 3.

[0093] In step S220 of some embodiments, the weight average value can only represent the average level of the full connection layer weight parameter, in order to more comprehensively describe the characteristics of the weight parameter, the variance indicating the dispersion degree of the weight parameter is introduced to comprehensively understand the weight parameter through the mean and variance, more effectively reduce the parameter size of the weight parameter, and ensure the stability of the model training and inference process. Specifically, the weight variance of the full connection layer is calculated according to the weight parameter of the full connection layer and the weight average value, and the weight variance value is obtained. The weight variance value of the kth full connection layer is represented as:

[0094]

[0095] wherein variance represents variance calculation; W k is the weight parameter of the kth full connection layer; is the weight variance value of the kth full connection layer.

[0096] In step S230 of some embodiments, the weight quantization parameter of the full connection layer is determined according to the weight average value and the weight variance value, and the weight quantization parameter is used to quantize the weight parameter of the full connection layer to reduce the data size of the weight parameter and simplify the complex calculation involving the weight parameter in the text generation process.

[0097] In step S240 of some embodiments, in order to reduce the model parameter size of the large language model and simplify the complex calculation of the large language model, for each full connection layer of the large language model, the weight parameter of the full connection layer is quantized according to the weight quantization parameter to obtain the target weight.

[0098] Through the above steps S210 to S240, the parameter size of the model parameter in the large language model can be reduced, and the complexity of the calculation related to the model parameter is reduced, the efficiency of the large language model training and inference is improved, and the efficiency of the text generation is improved.

[0099] Please refer to Figure 3 In some embodiments, step S230 can include but is not limited to steps S310 to S320:

[0100] Step S310, obtaining the weight mean square deviation according to the weight variance value;

[0101] Step S320, performing mean calculation on the weight average value and the weight mean square deviation to obtain the weight quantization parameter.

[0102] In step S310 of some embodiments, in order to more intuitively display the difference between the weight parameter and the weight average value, and improve the efficiency of weight quantization, the square root operation is performed on the weight variance value to obtain the weight mean square deviation.

[0103] In step S320 of some embodiments, the mean value calculation is performed on the weight average value and the weight variance to smooth the fluctuations of the quantization parameter, provide a more stable quantization parameter, and obtain the weight quantization parameter. The calculation formula of the weight quantization parameter is represented as:

[0104]

[0105] wherein r is the weight quantization parameter; is the weight variance value; is the weight average value.

[0106] Through the above steps S310 to S320, the weight quantization parameter can be obtained to quantize the weight parameter of the full connection layer based on the weight quantization parameter, reduce the model parameter size of the large language model, and ensure the stability of the model training and inference process, thereby improving the efficiency of text generation.

[0107] Referring to Figure 4 In some embodiments, step S240 can include but is not limited to steps S410 to S440:

[0108] In step S410, the weight parameter of the full connection layer is preliminarily quantized according to the weight quantization parameter to obtain an initial weight.

[0109] In step S420, the initial weight is rounded to obtain a first intermediate weight.

[0110] In step S430, the first intermediate weight is weight pruned to obtain a second intermediate weight.

[0111] In step S440, the second intermediate weight is quantized again according to the weight quantization parameter to obtain a target weight.

[0112] In step S410 of some embodiments, the weight quantization parameter and the weight parameter of the full connection layer are multiplied to preliminarily weight quantize the weight parameter, scale the weight parameter to a suitable value range, and obtain the initial weight.

[0113] In step S420 of some embodiments, when the large language model is used for text inference, too many decimal places of the initial weight will increase the calculation difficulty and affect the efficiency of text generation. When the storage medium stores and represents data, there is a fixed precision limit. In order to ensure that the initial weight does not exceed the precision limit and reduce the difficulty of text inference calculation, the initial weight is rounded to obtain the first intermediate weight.

[0114] In step S430 of some embodiments, the value of the first intermediate weight is too large, which consumes a large amount of storage resources and computing resources. In order to reduce the consumption of storage resources and computing resources in the model training and inference process, the first intermediate weight is clipped by a clipping function clip to limit the first intermediate weight to an integer value between [-1, 1] to obtain a second intermediate weight, so as to ensure the numerical stability of the large language model and prevent the extreme value of the weight from having a negative impact on the training and inference process of the large language model.

[0115] In step S440 of some embodiments, the reciprocal of the weight quantization parameter is obtained, and the second intermediate weight is multiplied by the reciprocal to restore the second intermediate weight to the value range of the original scale to obtain a target weight. Since the second intermediate weight can be 0, the number of quantized weight parameters, i.e., the target weight, is reduced, which reduces the model parameter size of the large language model. The calculation method of the target weight is represented as:

[0116]

[0117] where k represents the kth fully connected layer; is the target weight of the fully connected layer; W k is the weight parameter of the fully connected layer; * represents multiplication; r is the weight quantization parameter; round is the rounding operation; clip(-1, 1) represents that the integer value of W k *r after rounding is limited to [-1, 1].

[0118] By quantizing the weight parameters of the fully connected layer, the target weight is an integer between [-1, 0, 1], which can represent the target weight with ternary values, and a single weight parameter is quantized to 2 bits.

[0119] Through the above steps S410 to S440, the number of weight parameters in the fully connected layer can be reduced, thereby simplifying the complex operations caused by large-scale weight parameters, adjusting the weight parameters to a suitable value range, converting the weight parameters from high-precision floating-point numbers to low-precision integers, and reducing the parameter size and computational complexity of the large language model.

[0120] Please refer to Figure 5 In some embodiments, step S130 can include but is not limited to steps S510 to S520:

[0121] Step S510, filtering the text activation value and the preset constant to obtain an activation quantization parameter;

[0122] Step S520, quantizing the text activation value according to the activation quantization parameter to obtain a target activation value.

[0123] In step S510 of some embodiments, an activation quantization parameter is screened from the text activation value and a preset constant, and the activation quantization parameter is used to quantize the text activation value of the full connection layer to reduce the parameter size of the text feature parameter and accelerate the generation of the target text.

[0124] In step S520 of some embodiments, the activation quantization parameter is a scaling factor, and the text activation value is quantized according to the activation quantization parameter, and the text activation value is scaled to a suitable value range by using the scaling factor, which reduces the parameter size of the text activation value and obtains a target activation value.

[0125] Through the above steps S510 to S520, the parameter size of the text feature parameter, i.e., the text activation value, can be reduced, and the calculation complexity of the text feature parameter is reduced, and the full connection calculation of the full connection layer on the text feature is accelerated.

[0126] Please refer to Figure 6 In some embodiments, step S510 can include but is not limited to steps S610 to S640:

[0127] Step S610, selecting the maximum text activation value as a candidate activation value;

[0128] Step S620, comparing the candidate activation value with the preset constant;

[0129] Step S630, if the candidate activation value is greater than or equal to the preset constant, the candidate activation value is taken as the activation quantization parameter;

[0130] Step S640, if the candidate activation value is less than the preset constant, the preset constant is taken as the activation quantization parameter.

[0131] In step S610 of some embodiments, in order to ensure that all text activation values are within the quantization range, the maximum value of the text activation value is selected as the candidate activation value, so as to perform activation quantization according to the candidate activation value and reduce the quantization error.

[0132] In step S620 of some embodiments, in order to obtain a more accurate activation quantization parameter and avoid that the candidate activation value is too small to perform accurate activation quantization, a preset constant is set, and the candidate activation value is compared with the preset constant.

[0133] In steps S630 to S640 of some embodiments, in order to reduce the size of the model and improve the inference performance of the model, the maximum value of the candidate activation value and the preset constant is selected as the activation quantization parameter for the activation quantization of the full connection layer. If the candidate activation value is greater than or equal to the preset constant, the candidate activation value is taken as the activation quantization parameter. If the candidate activation value is less than the preset constant, the preset constant is taken as the activation quantization parameter.

[0134] Through the steps S610 to S640, the activation quantization parameter can be obtained to accurately quantize the text activation value based on the activation quantization parameter, and convert the text activation value from a high-precision floating point number to a low-precision integer, thereby reducing the storage requirement and the calculation amount of the model.

[0135] Please refer to Figure 7 In some embodiments, the step S520 can include but is not limited to steps S710 to S740:

[0136] Step S710, the text activation value is preliminarily quantized according to the activation quantization parameter to obtain an initial activation value;

[0137] Step S720, rounding the initial activation value to obtain a first intermediate activation value;

[0138] Step S730, the first intermediate activation value is clipped to obtain a second intermediate activation value;

[0139] Step S740, the second intermediate activation value is quantized again according to the activation quantization parameter to obtain a target activation value.

[0140] In step S710 of some embodiments, the text activation value is multiplied by the activation quantization parameter to scale the text activation value to a suitable value range, so as to preliminarily quantize the text activation value to obtain the initial activation value.

[0141] In step S720 of some embodiments, in order to simplify the calculation process of the activation value, the initial activation value is rounded to obtain the first intermediate activation value, so as to avoid the complexity of text activation calculation caused by too many decimal places.

[0142] In step S730 of some embodiments, the first intermediate activation value is clipped by the clip function to limit the first intermediate activation value to an integer value between [-128, 127] to obtain the second intermediate activation value, so as to convert the floating point calculation to integer calculation, thereby improving the efficiency of the activation value calculation.

[0143] In step S740 of some embodiments, the reciprocal of the activation quantization parameter is obtained, and the second intermediate activation value is multiplied by the reciprocal to restore the second intermediate activation value to the value range of the original scale to obtain the target activation value. Since the second intermediate activation value can be 0, the number of the quantized text activation value, i.e. the target activation value, is reduced, thereby simplifying the activation calculation process. The calculation method of the target activation value is represented as:

[0144]

[0145] Wherein, k represents the kth full connection layer; is the target activation value of the full connection layer; X k is the text activation value of the full connection layer; * represents multiplication operation; s is the activation quantization parameter; round is the rounding operation; clip(-128, 127) represents that X k is limited to [-128, 127] after rounding.

[0146] By quantizing the text activation value of the full connection layer, the target activation value is an integer between [-128, 127], and a single text activation value is quantized to an 8-bit signed integer type int8.

[0147] Through the above steps S710 to S740, the text activation value can be converted from a high-precision floating point number to a low-precision integer, reducing the complexity of the text activation calculation and reducing the size of the activation value, thereby improving the efficiency of the text generation.

[0148] In step S140 of some embodiments, the target text feature is subjected to root mean square layer normalization (Root Mean Square Layer Normalization) processing by the output layer to obtain normalized features, linear calculation is performed on the normalized features, the probability distribution of each word in the preset vocabulary is output, and the word with the highest probability distribution is selected to generate the target text.

[0149] The model training and inference in the Ascend 910A computing cluster container need to process data into the mindrecord file format. Taking the Alpaca dataset as an example, referring to the corresponding data processing interface of mindformers, fill in the problem sentence and the answer sentence according to the template, convert the templated problem sentence into input index input_tokens, convert the answer sentence into output index target_tokens, and save the input index and the output index into the mindrecord format. Use the convert_weight script in mindformers to convert the large language model into a weight file matched with the Ascend architecture. Test the performance of the large language model on the Alpaca dataset, which includes the Alpaca training set and the Alpaca test set. (1) Use the large language model for inference, and get the Perplexity (PPL) of the model in the Alpaca test set as a, PPL is used to measure the probability prediction ability of the model on the sentence in the test set, and the smaller the PPL is, the better the performance of the model is. (2) Set up a non-fine-tuning quantization experiment, still use the model parameters of the large language model, update the calculation method of the fully connected layer in the large language model through weight quantization and activation quantization, and use the quantized calculation method for inference, and get the PPL of the model in the Alpaca test set as b. (3) Set up a micro-quantization experiment, fine-tune the quantized large language model through the Alpaca training set, train only 2 Epoch rounds in the fine-tuning, and use 8 graphics cards for parallel training. Test the trained large language model to get the PPL of the model in the Alpaca test set as c.

[0150] The experimental results show that b is much larger than a and c, and a and c are almost equal, which indicates that the full connection layer quantization of the embodiment of the application reduces the data precision of model calculation, reduces data storage and power consumption while ensuring the correctness of model inference, and proves the effectiveness of the large language model lightening method.

[0151] Please refer to Figure 8 The embodiment of the application also provides a text generation device based on large model quantization, which is applied to the Ascend computing architecture and can realize the above-mentioned text generation method based on large model quantization. The text generation device based on large model quantization comprises:

[0152] The acquisition module 810 is configured to acquire an original text.

[0153] The input module 820 is configured to input the original text into a target large language model, and the target large language model comprises a plurality of feature extraction layers which are cascaded in sequence, and each feature extraction layer comprises a full connection layer.

[0154] The feature extraction module 830 is configured to perform layer-by-layer incremental feature extraction on the text features corresponding to the original text through a plurality of feature extraction layers in the target large language model, and determine the candidate text features output by the last layer of feature extraction layer as the target text features.

[0155] The feature extraction layer at each layer is configured to perform attention calculation on the input text features to obtain attention text features, perform activation processing on the attention text features to obtain text activation values, perform quantization processing on the text activation values to obtain target activation values, and output the candidate text features at the current layer according to the target activation values and target weights obtained by quantizing the weight parameters of the corresponding full connection layer.

[0156] The process of layer-by-layer incremental feature extraction is to use the candidate text features extracted at the previous layer as the feature basis data for feature extraction at the next layer.

[0157] The text generation module 840 is configured to generate the target text based on the target text features.

[0158] The specific implementation of the text generation device based on large model quantization is basically the same as that of the above-mentioned specific embodiments of the text generation method based on large model quantization, and will not be repeated here.

[0159] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to realize the above-mentioned text generation method based on large model quantization. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0160] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0161] The processor 910 can be implemented in a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0162] The memory 920 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 920 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 920 and are called and executed by the processor 910 to implement the text generation method based on large model quantization according to the embodiments of the present application;

[0163] The input / output interface 930 is configured to realize information input and output.

[0164] The communication interface 940 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0165] The bus 950 is configured to transmit information between various components (for example, the processor 910, the memory 920, the input / output interface 930, and the communication interface 940) of the device.

[0166] The processor 910, the memory 920, the input / output interface 930, and the communication interface 940 are connected to each other through the bus 950 to realize the communication connection between the devices.

[0167] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the text generation method based on large model quantization.

[0168] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0169] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0170] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0171] The apparatus embodiments described above are merely illustrative, and units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0172] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0173] The terms "first", "second", "third", "fourth" and the like used in the description of the present application and the above-described figures (if any) are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0174] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0175] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0176] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0177] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0178] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0179] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, which are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A large model-based quantization text generation method, characterized in that, The method is applied to the Ascend computing architecture, and the method comprises the following steps: obtaining an original text; inputting the original text into a target large language model, the target large language model comprising a plurality of feature extraction layers connected in sequence, each feature extraction layer comprising a full connection layer; based on the text features corresponding to the original text, performing layer-by-layer incremental feature extraction through the plurality of feature extraction layers connected in sequence in the target large language model, and determining the candidate text features output by the last level of the feature extraction layer as target text features; wherein, for each level, the feature extraction layer is used to perform attention calculation on the input text features to obtain attention text features, and perform activation processing on the attention text features to obtain text activation values, quantize the text activation values to obtain target activation values, and output the candidate text features of the current level according to the target activation values and the target weights obtained by quantizing the weight parameters of the corresponding full connection layer; wherein, the process of layer-by-layer incremental feature extraction is to take the candidate text features extracted in the previous level as the feature basis data for feature extraction in the next level; generating a target text based on the target text features; the target weights are calculated according to the following steps: performing mean calculation on the weight parameters of the full connection layer to obtain a weight average value, calculating the variance of the weight parameters of the full connection layer according to the weight average value to obtain a weight variance value, determining a weight quantization parameter according to the weight average value and the weight variance value, and quantizing the weight parameters of the full connection layer according to the weight quantization parameter to obtain the target weights; the quantization processing of the weight parameters of the full connection layer according to the weight quantization parameter to obtain the target weights comprises: performing preliminary quantization on the weight parameters of the full connection layer according to the weight quantization parameter to obtain initial weights, rounding the initial weights to obtain first intermediate weights, performing weight clipping on the first intermediate weights to obtain second intermediate weights, and performing re-quantization on the second intermediate weights according to the weight quantization parameter to obtain the target weights; the determination of the weight quantization parameter according to the weight average value and the weight variance value comprises: obtaining a weight mean square difference according to the weight variance value, and performing mean calculation on the weight average value and the weight mean square difference to obtain the weight quantization parameter.

2. The method of claim 1, wherein, the quantization processing of the text activation values to obtain target activation values comprises: screening the text activation values and a preset constant to obtain an activation quantization parameter; quantizing the text activation values according to the activation quantization parameter to obtain the target activation values.

3. The method of claim 2, wherein, the quantization processing of the text activation values according to the activation quantization parameter to obtain the target activation values comprises: performing preliminary quantization on the text activation values according to the activation quantization parameter to obtain initial activation values; rounding the initial activation values to obtain first intermediate activation values; performing activation clipping on the first intermediate activation values to obtain second intermediate activation values; According to the activation quantization parameter, the second intermediate activation value is re-quantized to obtain the target activation value.

4. The method of claim 2, wherein, The filtering the text activation value and the preset constant to obtain an activation quantization parameter comprises: selecting the maximum text activation value as a candidate activation value; comparing the candidate activation value with the preset constant; if the candidate activation value is greater than or equal to the preset constant, the candidate activation value is taken as the activation quantization parameter; if the candidate activation value is less than the preset constant, the preset constant is taken as the activation quantization parameter. 5.A text generation apparatus based on large model quantization, characterized by, The device is applied to an Ascend computing architecture, and the device comprises: an acquisition module configured to acquire an original text; an input module configured to input the original text into a target large language model, the target large language model comprising a plurality of feature extraction layers that are sequentially cascaded, each feature extraction layer comprising a full connection layer; a feature extraction module configured to perform layer-by-layer incremental feature extraction on a text feature corresponding to the original text based on the plurality of feature extraction layers that are sequentially cascaded in the target large language model, and determine a candidate text feature output by the feature extraction layer of the last layer as a target text feature; wherein the feature extraction layer of each layer is configured to perform attention calculation on an input text feature to obtain an attention text feature, perform activation processing on the attention text feature to obtain a text activation value, perform quantization processing on the text activation value to obtain a target activation value, and output a candidate text feature of a current layer according to a target weight obtained by quantizing a weight parameter of the corresponding full connection layer; wherein the process of the layer-by-layer incremental feature extraction is to take the candidate text feature extracted by a previous layer as feature basis data for feature extraction of a next layer; a text generation module configured to generate a target text based on the target text feature; the target weight is calculated according to the following steps: performing mean value calculation on the weight parameter of the full connection layer to obtain a weight average value, calculating the variance of the weight parameter of the full connection layer according to the weight average value to obtain a weight variance value, determining a weight quantization parameter according to the weight average value and the weight variance value, and performing quantization processing on the weight parameter of the full connection layer according to the weight quantization parameter to obtain the target weight; the quantization processing on the weight parameter of the full connection layer according to the weight quantization parameter to obtain the target weight comprises: performing preliminary quantization on the weight parameter of the full connection layer according to the weight quantization parameter to obtain an initial weight, performing rounding on the initial weight to obtain a first intermediate weight, performing weight clipping on the first intermediate weight to obtain a second intermediate weight, and performing re-quantization on the second intermediate weight according to the weight quantization parameter to obtain the target weight; the determination of the weight quantization parameter according to the weight average value and the weight variance value comprises: obtaining a weight mean square error according to the weight variance value, and performing mean value calculation on the weight average value and the weight mean square error to obtain the weight quantization parameter.

6. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. The computer program is executed by the processor to implement the method in any one of claims 1 to 4.

Images