Text generation method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a text generation method, device and equipment and a storage medium, and relate to the technical field of artificial intelligence. The method obtains a training text set and model parameters, inputs a text sample into an initial large language model to perform text prediction, performs forward propagation using the model parameters to obtain predicted text, obtains a loss function based on the predicted text, calculates a gradient value of the loss function with respect to the model parameters, updates a momentum parameter using a decay parameter and the gradient value, updates a weight parameter using a learning rate, a weight decay rate and the updated momentum parameter, and iterates the weight parameter multiple times until a target large language model is obtained. Then, the obtained to-be-processed text is input into the target large language model to obtain target generated text. In the training process, the momentum parameter is used as an intermediate state parameter, the intermediate state parameter is reduced, the memory occupation is reduced, the resource utilization rate of the processor is improved, the training cost of the large language model is reduced, and the application range of the large language model is expanded.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to text generation methods, apparatus, devices and storage media. Background Technology

[0002] Large language models are a neural network-based natural language processing technique that can be used in applications such as text generation, machine translation, and dialogue systems. Specifically, large language models learn from large amounts of text data, attempting to predict the probability of the next word or character appearing, thereby generating text similar to the training data.

[0003] In related technologies, the core of large language models is a long short-term memory neural network or a modified transformer neural network structure. These models use the AdamW optimizer to adjust the model weights during training. However, the AdamW optimizer requires a large number of intermediate state parameters during the update process, which consumes a lot of GPU memory and has a high computational cost. Summary of the Invention

[0004] The main objective of this application is to propose a text generation method, apparatus, device, and storage medium to reduce the GPU memory usage and computational overhead during the training process of large language models, thereby improving the training efficiency of large language models.

[0005] To achieve the above objectives, a first aspect of this application proposes a text generation method, comprising:

[0006] Obtain the training text set and the model parameters of the initial large language model. The training text set includes text samples and generated labels. The model parameters include: weight parameters, momentum parameters, learning rate, weight decay rate, and decay parameters.

[0007] The text sample is input into the initial large language model for text prediction, and the model parameters are used for forward propagation to obtain the predicted text of the text sample.

[0008] The loss function is obtained based on the predicted text and the generated labels, and the gradient value of the loss function with respect to the model parameters is calculated. The momentum parameter is updated using the decay parameter and the gradient value, and the weight parameter is updated using the learning rate, the weight decay rate and the updated momentum parameter. The weight parameter is iterated multiple times until the initial large language model is trained and the target large language model is obtained.

[0009] Obtain the text to be processed and input it into the target large language model to obtain the target generated text.

[0010] In some embodiments, updating the momentum parameter using the decay parameter and the gradient value includes:

[0011] The first value is calculated based on the decay parameter and the momentum parameter of the previous moment;

[0012] The second value is calculated based on the attenuation parameter and the gradient value;

[0013] The momentum parameter at the current moment is obtained based on the first value and the second value, and the momentum parameter at the current moment is used as the updated momentum parameter.

[0014] In some embodiments, updating the weight parameters using the learning rate, the weight decay rate, and the updated momentum parameters includes:

[0015] The third value is calculated based on the weight parameters from the previous moment and the weight decay rate.

[0016] The fourth value is calculated by using the third value and the momentum parameter at the current moment as inputs to the sign function;

[0017] Calculate the fifth value based on the learning rate and the fourth value;

[0018] The difference between the weight parameter at the previous moment and the fifth value is calculated to obtain the weight parameter at the current moment, and the weight parameter at the current moment is used as the updated weight parameter.

[0019] In some embodiments, the weight parameters include the bias value of the linear layer, the scaling value of the normalized layer, the translation value of the normalized layer, and other parameters; the calculation process of the learning rate includes the following steps:

[0020] If the weight parameter is the bias value, the scaling value, or the translation value, then the learning rate is obtained based on the global relative learning rate;

[0021] If the weight parameter is one of the other parameters, the learning rate is obtained based on the root mean square of the weight parameter and the global relative learning rate.

[0022] In some embodiments, the calculation process for the attenuation rate includes the following steps:

[0023] If the weight parameter is the bias value, the scaling value, or the translation value, then the attenuation rate is set to zero.

[0024] If the weight parameter is one of the other parameters, then the decay rate is set to a preset constant greater than zero.

[0025] In some embodiments, the model parameters further include a gradient accumulation metric. Updating the momentum parameter using the decay parameter and the gradient value, and updating the weight parameter using the learning rate, the weight decay rate, and the updated momentum parameter, includes:

[0026] A sixth value is calculated using the current momentum parameter, the decay parameter, and the gradient accumulation index, and a seventh value is calculated based on the gradient value, the decay parameter, and the number of weights in the weight parameter. The updated momentum parameter is obtained based on the sum of the sixth value and the seventh value.

[0027] Multiply the gradient accumulation index and the fifth value to obtain the eighth value;

[0028] The updated weight parameter is obtained by calculating the difference between the current weight parameter and the eighth value.

[0029] In some embodiments, the calculation process of the gradient accumulation index includes:

[0030] Get the iteration time of the current iteration;

[0031] If the iteration time is divisible by the number of weights, the gradient accumulation index is a first preset value; otherwise, the gradient accumulation index is a second preset value.

[0032] To achieve the above objectives, a second aspect of this application provides a text generation apparatus, comprising:

[0033] Data acquisition module: used to acquire the training text set and the model parameters of the initial large language model. The training text set includes text samples and generated labels. The model parameters include: weight parameters, momentum parameters, learning rate, weight decay rate and decay parameters.

[0034] Text prediction module: used to input the text sample into the initial large language model for text prediction, and use the model parameters for forward propagation to obtain the predicted text of the text sample;

[0035] Parameter update module: used to obtain a loss function based on the predicted text and the generated labels, calculate the gradient value of the loss function with respect to the model parameters, update the momentum parameter using the decay parameter and the gradient value, update the weight parameter using the learning rate, the weight decay rate and the updated momentum parameter, iterate the weight parameter multiple times until the initial large language model is trained and the target large language model is obtained;

[0036] Text generation module: used to acquire the text to be processed and input the text to be processed into the target large language model to obtain the target generated text.

[0037] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0038] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0039] The text generation method, apparatus, device, and storage medium proposed in this application involve acquiring a training text set and model parameters of an initial large language model, inputting text samples into the initial large language model for text prediction, performing forward propagation using the model parameters to obtain the predicted text of the text samples, obtaining a loss function based on the predicted text and generated labels, calculating the gradient value of the loss function with respect to the model parameters, updating the momentum parameters using decay parameters and gradient values, and updating the weight parameters using the learning rate, weight decay rate, and the updated momentum parameters, iterating the weight parameters multiple times until the initial large language model is trained to obtain a target large language model, acquiring the text to be processed, and inputting the text to be processed into the target large language model to obtain the target generated text. In this application embodiment, the large language model uses momentum parameters as intermediate state parameters during training. Compared with the AdamW optimizer in related technologies, this reduces intermediate state parameters, lowers memory usage, improves processor resource utilization, lowers the training cost of the large language model, and thus expands the applicability of the large language model. Attached Figure Description

[0040] Figure 1 This is a flowchart of the text generation method provided in the embodiments of this application.

[0041] Figure 2 This is a flowchart of updating momentum parameters using decay parameters and gradient values, provided in an embodiment of this application.

[0042] Figure 3 This is a flowchart of updating weight parameters using the learning rate and the updated momentum parameter, provided in an embodiment of this application.

[0043] Figure 4 A flowchart illustrating the calculation of the learning rate provided in this application embodiment.

[0044] Figure 5 A flowchart illustrating the calculation of attenuation rate provided in an embodiment of this application.

[0045] Figure 6This is a flowchart of the update process for weight parameters under gradient accumulation provided in the embodiments of this application.

[0046] Figure 7 This is a structural block diagram of a text generation device provided in another embodiment of this application.

[0047] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0051] First, let's analyze some of the terms used in this application:

[0052] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0053] Large language models are a neural network-based natural language processing technique that can be used in applications such as text generation, machine translation, and dialogue systems. Specifically, large language models learn from large amounts of text data, attempting to predict the probability of the next word or character appearing, thereby generating text similar to the training data.

[0054] In related technologies, the core of large language models is a long short-term memory neural network or a modified transformer neural network structure. These models use the AdamW optimizer to adjust the model weights during training.

[0055] For example, the update process of the AdamW optimizer is as follows:

[0056]

[0057] Where, m t_Adam G represents the momentum parameter of the AdamW optimizer. t_Adam β represents the gradient value of the loss function of the AdamW optimizer. 1_Adam β 2_Adam These represent different decay parameters of the AdamW optimizer. ε _Adam u t_Adam θ represents the intermediate value of the AdamW optimizer. t_Adam η represents the weight parameters of the AdamW optimizer. t_Adam λ represents the learning rate of the AdamW optimizer. _Adam represents the decay rate of the AdamW optimizer, t represents the current iteration time, and t-1 represents the previous iteration time.

[0058] It is evident that the AdamW optimizer uses a large number of intermediate state parameters during the update process, resulting in high memory usage. This leads to significant computational overhead when training large-scale deep learning models, with the optimizer state typically occupying 75% of the memory. This limits the scale and training speed of large language models.

[0059] Based on this, embodiments of this application provide a text generation method, apparatus, device, and storage medium. In the large language model of this application, momentum parameters are used as intermediate state parameters during the training process. Compared with the AdamW optimizer in related technologies, the number of intermediate state parameters is reduced, the memory usage is reduced, the resource utilization of the processor is improved, the training cost of the large language model is reduced, and the applicability of the large language model is expanded.

[0060] This application provides a text generation method, apparatus, device, and storage medium, which are specifically described through the following embodiments. First, the text generation method in this application is described.

[0061] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0062] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0063] The text generation method provided in this application relates to the field of artificial intelligence technology. This text generation method can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client that supports text generation; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The text generation method can be executed by the terminal or the server, or by the terminal and the server working together.

[0064] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or a network, etc., and this embodiment does not impose any limitations.

[0065] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0066] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0067] The text generation method in the embodiments of this application is described below.

[0068] Figure 1 This is an optional flowchart of the text generation method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S140. It is also understood that this embodiment... Figure 1 The order of steps S110 to S140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0069] Step S110: Obtain the training text set and the model parameters of the initial large language model.

[0070] In one embodiment, the training text set includes text samples and generated labels, wherein the text samples are the materials used for text generation, the large language model generates text based on the text samples, and the generated labels are used to correct the generation results of the large language model and are the expected generated content corresponding to the text samples.

[0071] In one embodiment, the model parameters corresponding to the current time t include: weight parameters θ t Momentum parameter m t Learning rate η t Weight decay rate λ t and the attenuation parameter β.

[0072] Wherein, the weight parameter θ t The first parameter represents the weight information contained in different layers of a large language model. For example, if a large language model includes linear layers and normalization layers, the weight parameters include the bias value of the linear layer, the scaling and translation values ​​of the normalization layer, and other parameters. The bias value of the linear layer is used to adjust the offset of the output result. During computation, a bias value is added to each element of the input tensor. The scaling and translation values ​​of the normalization layer are used to scale and translate the input tensor to adjust the mean and variance of the features to better adapt to different data distributions. Other parameters refer to parameters in the large language model other than the three parameters mentioned above.

[0073] Wherein, the learning rate η t The momentum parameter m controls the magnitude of model parameter updates during each iteration, determining the step size traversed in the weight space. t During each iteration, the momentum parameter accumulates information about the current gradient, retaining the influence of historical gradients to avoid getting trapped in local minima and accelerating convergence. The weight decay rate λ tTo control overfitting, a regularization term is introduced into the loss function to constrain the weight parameters of the large language model. This regularization term causes the large language model to tend to choose smaller weight values ​​during training, thereby reducing the complexity of the large language model and reducing the risk of overfitting. The decay parameter β is used to reduce the learning rate, allowing the large language model to still effectively update parameters in the later stages of training.

[0074] Before training begins, the weight parameters θ are first... t Initialization is performed to obtain the initial large language model.

[0075] Step S120: Input the text sample into the initial large language model for text prediction, and use the model parameters for forward propagation to obtain the predicted text of the text sample.

[0076] In one embodiment, each iteration has corresponding model parameters. For example, the iteration time t corresponds to the model parameter θ. t The feature information of the text samples is propagated forward in the large language model using model parameters until the output predicted text is used as the text prediction result under the model parameters.

[0077] In large language models, forward propagation refers to the computation of input data through the layers of a neural network to obtain the model's predicted output. Specifically, the forward propagation computation steps typically include the following parts: 1) Input layer: Representing the input data features in a form that the neural network can process. 2) Hidden layer: The intermediate layer of the neural network, containing multiple neurons. Each neuron weights and sums the outputs of the previous layer, then processes the sum through an activation function to obtain the output. 3) Weighted summation: In each neuron, the input value is multiplied by its corresponding weight, and then all weighted input values ​​are summed. This summation process can be represented by matrix multiplication and usually includes a bias. 4) Activation function: The result of the weighted summation is input into the activation function. The activation function introduces non-linear properties, allowing the neural network to learn non-linear relationships. Commonly used activation functions include ReLU, Sigmoid, and Tanh. 5) Output layer: The number of neurons in the output layer usually matches the number of categories in the task or the output dimension of a regression task. Through these steps, the forward propagation process can compute the predicted text of a large language model.

[0078] Step S130: Obtain the loss function based on the predicted text and generated labels, calculate the gradient value of the loss function with respect to the model parameters, update the momentum parameter using the decay parameter and gradient value, and update the weight parameter using the learning rate and the updated momentum parameter. Iterate the weight parameter multiple times until the initial large language model is trained and the target large language model is obtained.

[0079] In one embodiment, after obtaining the predicted text, a loss function for the current time t needs to be derived based on the predicted text and generated labels. This loss function measures the difference between the predicted text and generated labels of the large language model and is typically minimized during training. In this embodiment, the loss function is the cross-entropy loss function. It is understood that at the current time t, the weight parameters have not yet been adjusted, and the loss function is based on the weight parameters θ from the previous time t-1. t-1 The calculated loss function is therefore expressed as L(θ). t-1 Next, the gradient value g of the loss function is calculated under the weight parameters. t , is represented as:

[0080]

[0081] In one embodiment, after obtaining the gradient value, a weight parameter update process is performed. (Refer to...) Figure 2 , Figure 2 This is a flowchart of updating momentum parameters using decay parameters and gradient values, provided in an embodiment of this application, including the following steps S210 to S230:

[0082] Step S210: Calculate the first value based on the decay parameter and the momentum parameter of the previous moment.

[0083] Step S220: Calculate the second value based on the attenuation parameter and gradient value.

[0084] Step S230: Obtain the momentum parameter at the current moment based on the first value and the second value, and use the momentum parameter at the current moment as the updated momentum parameter.

[0085] The first value is represented as:

[0086] βm t-1

[0087] Where β represents the attenuation parameter, m t-1 This represents the momentum parameter at the previous moment.

[0088] In comparison, the AdamW optimizer requires two decay parameters β. 1_Adam and β 2_Adam In this embodiment, only one attenuation parameter β needs to be set, without the need to cache different attenuation parameters, thus avoiding performance loss due to improper selection of the two attenuation parameters.

[0089] In one embodiment, the following can be set:

[0090]

[0091] Assume β 1_Adam =0.9, β 2_Adam=0.99, then β =0.945. Assume β 1_Adam =0.95, β 2_Adam =0.98, then β = 0.965. It is understandable that β... 1_Adam and β 2_Adam It can be set based on experience values.

[0092] The second value is expressed as:

[0093] (1-β)g t

[0094] The updated momentum parameter is expressed as follows:

[0095] m t =βm t-1 +(1-β)g t

[0096] Where, m t This represents the momentum parameter at the current moment.

[0097] As can be seen, compared to the AdamW optimizer, the embodiments of this application introduce a momentum parameter, eliminating the need for caching during the update process. ε _Adam u t_Adam It saves more video memory by processing intermediate data.

[0098] In one embodiment, reference is made to Figure 3 , Figure 3 This is a flowchart of updating weight parameters using the learning rate and the updated momentum parameter, provided in an embodiment of this application, including the following steps S310 to S340:

[0099] Step S310: Calculate the third value based on the weight parameters and weight decay rate of the previous time step.

[0100] Step S320: Using the third value and the momentum parameter at the current moment as inputs to the sign function, calculate the fourth value.

[0101] Step S330: Calculate the fifth value based on the learning rate and the fourth value.

[0102] Step S340: Calculate the difference between the weight parameter at the previous time step and the fifth value to obtain the weight parameter at the current time step, and use the weight parameter at the current time step as the updated weight parameter.

[0103] In one embodiment, the third value is represented as:

[0104] λθ t-1

[0105] Where λ represents the weight decay rate, θ t-1 This represents the weight parameters at the previous time step.

[0106] The fourth value is represented as:

[0107] sign(m t +λθ t-1 )

[0108] Here, sign() represents the sign function, with 1 for positive numbers and -1 for negative numbers.

[0109] The fifth value is represented as:

[0110] η t (sign(m t +λθ t-1 ))

[0111] Where, η t This represents the learning rate at the current moment.

[0112] Therefore, the updated weight parameters are expressed as:

[0113] θ t =θ t-1 -η t (sign(m t +λθ t-1 ))

[0114] Where, θ t This represents the weight parameters at the current time.

[0115] In one embodiment, reference is made to Figure 4 , Figure 4 The flowchart for calculating the learning rate provided in this application embodiment specifically includes steps S410 to S420:

[0116] Step S410: If the weight parameter is a bias value, scaling value, or translation value, then the learning rate is obtained based on the global relative learning rate.

[0117] Step S420: If the weight parameters are other parameters, the learning rate is obtained based on the root mean square of the weight parameters and the global relative learning rate.

[0118] In one embodiment, other parameters are typically the kernel matrix of a linear layer. These parameters are characterized by being multiplied by vectors as matrices. In this embodiment, the global relative learning rate α is used to update the weights, depending on the weight parameters to be updated. t Different learning rates are selected, and the global relative learning rate α is used here. t It can be set according to actual needs.

[0119] Therefore, the learning rate η t Represented as:

[0120]

[0121]

[0122] Where k represents the number of weights in the weight parameters, bias represents the bias value, beta represents the scaling value, and gamma represents the translation value.

[0123] This application embodiment separates the scale of the weight parameters by setting the learning rate as described above, and uses a global relative learning rate α, which is insensitive to the scale of large language models. t To adjust the learning rate.

[0124] In one embodiment, reference is made to Figure 5 , Figure 5 The flowchart for calculating the attenuation rate provided in the embodiments of this application specifically includes steps S510 to S520:

[0125] Step S510: If the weight parameter is an offset value, scaling value, or translation value, then set the decay rate to zero.

[0126] Step S520: If the weight parameter is another parameter, then set the decay rate to a preset constant greater than zero.

[0127] In one embodiment, from the perspective of prior distribution, the weight decay is a Gaussian prior of the weight parameters, and the decay rate is inversely proportional to the variance of the weight parameters. However, the variance of the bias value, scaling value, or translation value is larger than the variance of the kernel matrix. Therefore, in this embodiment, the decay rate corresponding to the bias value, scaling value, or translation value is set to be smaller.

[0128] Therefore, the attenuation rate is expressed as:

[0129]

[0130] From the above, the optimizer representation of the large language model in the training process of this application embodiment can be derived as follows:

[0131]

[0132] Wherein, Leopard represents the optimizer in this application embodiment.

[0133] As can be seen, compared to the AdamW optimizer in related technologies, the Leopard optimizer in this embodiment introduces momentum parameters and weight decay, using fewer intermediate state parameters and saving cache. It also avoids the division and square root operations, which have the highest computational cost in the AdamW optimizer, thus reducing computational overhead. Furthermore, due to the presence of momentum parameters and weight decay, convergence to the optimal solution is accelerated during training, avoiding the problem of continuously increasing weight parameters, improving the generalization performance of large language models, preventing overfitting, and enhancing model stability. Using only momentum parameters only reduces computational transmission costs but cannot guarantee optimization performance.

[0134] In one embodiment, considering that when computing power is limited, the batch size can be increased through gradient accumulation during the training of a large language model. For the AdamW optimizer, three new sets of parameters are required. However, the optimizer in this embodiment only uses momentum and weight parameters for its update, so gradient accumulation can be performed using the following gradient accumulation metric. At this time, refer to... Figure 6 , Figure 6 This is a flowchart of the update of weight parameters under gradient accumulation provided in the embodiments of this application, specifically including the following steps S610 to S630:

[0135] Step S610: Calculate the sixth value using the current momentum parameter, decay parameter and gradient accumulation index, and calculate the seventh value based on the gradient value, decay parameter and the number of weights in the weight parameter. The updated momentum parameter is obtained based on the sum of the sixth value and the seventh value.

[0136] The calculation process for the gradient accumulator index includes: obtaining the iteration time of the current iteration; if the iteration time is divisible by the number of weights, the gradient accumulator index is a first preset value; otherwise, it is a second preset value. For example, if the first preset value is 1 and the second preset value is 0, the gradient accumulator index is expressed as:

[0137]

[0138] Where, χ t / k denoted by gradient accumulation index, k represents the number of weights, t represents the iteration time, and mod represents the modulo operation.

[0139] At this point, the sixth value is represented as:

[0140] ((β-1)χ (t-1) / k +1))m t-1

[0141] The seventh value is represented as:

[0142]

[0143] The updated momentum parameter is expressed as follows:

[0144]

[0145] Step S620: Multiply the gradient accumulation index and the fifth value to obtain the eighth value.

[0146] Step S630: Calculate the difference between the current weight parameter and the eighth value to obtain the updated weight parameter.

[0147] The eighth value is represented as:

[0148] χ t / k η t (sign(m t +λθ t-1 )

[0149] The updated weight parameters are expressed as follows:

[0150] θ t =θ t-1 -χ t / k η t (sign(m t +λθ t-1 )

[0151] Therefore, based on the above, the optimizer for gradient accumulation during the training process of the large language model in this application embodiment can be represented as follows:

[0152]

[0153] As can be seen, the optimizer in this embodiment of the application, when performing gradient accumulation, is equivalent to only modifying the decay parameter β and the learning rate η. t It can automatically accumulate gradients without increasing the cost of video memory.

[0154] Related technologies use mixed precision for data storage during training. This means that half-precision FP16 is used to store computational data in the model computation part, while single-precision FP32 is used to store model parameters. The reason for using FP32 for model parameters is that if the update amount of model parameters is too small during the update process, it will overflow the representation range of FP16, causing some model parameters to remain stagnant for a long time, affecting the training progress of large language models, or even preventing normal training. However, in this embodiment, the optimizer performs a signed operation on the update amount of the weight parameters when updating the momentum parameter. Therefore, this embodiment can use half-precision for data storage throughout the training process. During storage, the gradient value gt will not overflow the representation range of FP16; and the momentum parameter mt is simply a moving average of the gradient values. If the gradient value does not overflow, the momentum parameter will not overflow either. The result of the signed operation can only be ±1, so it will not overflow. Therefore, the optimizer in this embodiment can be represented within the FP16 range throughout the entire update process, and can be trained directly with full FP16 precision without worrying about overflow issues. This results in lower memory usage during training. Meanwhile, because FP16 computation has a higher computational density than FP32 computation, it can achieve higher parallel computing capabilities, and the optimizer can be trained faster.

[0155] Repeat the above steps, iterating the weight parameters multiple times until the iteration termination condition is met, thus completing the training of the initial large language model and obtaining the target large language model. The iteration termination condition includes: the number of iterations reaching a preset number, or the model performance of the large language model after iteration reaching a preset performance requirement. This embodiment does not specifically limit the iteration termination condition.

[0156] In one embodiment, the performance of the Leopard optimizer provided in this application is verified. This primarily involves comparison with the AdamW optimizer in related technologies, using large language model frameworks of different sizes as examples. These large language models include T5Base (220M), Large (770M), and the largest 11B T5 model. The datasets used are: CoLA dataset, SST-2 dataset, MRPC dataset, STS-B dataset, QQP dataset, MNLI dataset, and QNLI dataset.

[0157] Referring to Table 1, which shows the performance of the optimizer in the embodiments of this application.

[0158] Model Optimizer CoLA SST-2 MRPC STS-B QQP MNLI QNLI RTE Base AdamW 60.97 95.20 92.49 90.60 89.33 86.87 93.80 81.69 Base Leopard 61.17 95.28 92.62 90.80 89.62 87.77 93.95 85.76 Large AdamW 63.99 96.20 93.60 91.79 90.18 89.72 94.55 89.27 Large Leopard 65.22 96.32 94.76 91.88 90.33 89.89 94.96 90.34 11B AdamW 69.60 97.12 93.85 92.67 90.55 92.07 96.51 92.52 11B Leopard 71.42 97.23 94.68 93.24 90.77 91.98 96.78 93.96

[0159] As shown in Table 1, on average, the Leopard optimizer in this embodiment outperforms the Adam optimizer across all three model scales.

[0160] Referring to Table 2, which shows another performance result of the optimizer in the embodiments of this application, Table 2 presents a performance comparison between the Leopard optimizer and the AdamW optimizer in related technologies when training ResNet-50 and vitb / 16 models on ImageNet. The data augmentation part uses (RandAug and Mixup). It can be seen that the Leopard optimizer and the AdamW optimizer achieve comparable performance; the advantage of the Leopard optimizer becomes more pronounced as the dataset grows.

[0161] Model Task Data Aug AdamW Leopard ResNet-50 ImageNet RandAug+Mixup 78.89 78.63 ResNet-50 ImageNetReaL RandAug+Mixup 84.61 84.57 ResNet-50 ImageNetV2 RandAug+Mixup 66.73 66.69 ViT-B / 16 ImageNet RandAug+Mixup 80.12 80.07 ViT-B / 16 ImageNetReaL RandAug+Mixup 85.46 85.54 ViT-B / 16 ImageNetV2 RandAug+Mixup 68.14 68.49

[0162] As shown in Tables 1 and 2 above, the optimizer of this embodiment achieves the same performance as the AdamW optimizer, but with significantly reduced GPU memory usage while maintaining efficient training speed and convergence. Under the same GPU resource conditions, the large language model trained in this embodiment is larger, training is faster, convergence is better, and resource utilization is higher during training.

[0163] Step S140: Obtain the text to be processed and input it into the target large language model to obtain the target generated text.

[0164] After training the initial large language model through the above process, the target large language model is obtained. At this time, the text to be processed for text generation is obtained, and the text to be processed is input into the target large language model to obtain the target generated text that meets the requirements.

[0165] The optimizer used in the training process of the target large language model in this application embodiment reduces intermediate state parameters compared to optimizers in related technologies, making the calculation process simpler. It maintains both memory efficiency and training speed, ensuring the large language model converges in a shorter time. Furthermore, while maintaining training effectiveness, it simplifies the update rules based on momentum parameters, enabling stable and efficient training even in full and half-precision training, thus solving the training limitations encountered in mixed-precision training. Simultaneously, the optimizer incorporates gradient accumulation, enabling gradient accumulation when processing large-scale data, thereby allowing for larger batch training and significantly reducing memory usage during large language model training. This allows for smooth training even in memory-constrained environments, achieving memory usage optimization without the need for additional parameters.

[0166] The technical solution provided in this application involves obtaining a training text set and the model parameters of an initial large language model, inputting text samples into the initial large language model for text prediction, performing forward propagation using the model parameters to obtain the predicted text of the text samples, obtaining a loss function based on the predicted text and generated labels, calculating the gradient value of the loss function with respect to the model parameters, updating the momentum parameters using the decay parameters and gradient values, and updating the weight parameters using the learning rate, weight decay rate, and the updated momentum parameters, iterating the weight parameters multiple times until the initial large language model is trained to obtain the target large language model, obtaining the text to be processed, and inputting the text to be processed into the target large language model to obtain the target generated text. In this application embodiment, the large language model uses momentum parameters as intermediate state parameters during training. Compared with the AdamW optimizer in related technologies, this reduces intermediate state parameters, lowers memory usage, improves processor resource utilization, lowers the training cost of the large language model, and thus expands the applicability of the large language model.

[0167] This application also provides a text generation apparatus that can implement the above-described text generation method, see reference. Figure 7 The device includes:

[0168] Data acquisition module 710: Used to acquire the training text set and the model parameters of the initial large language model. The training text set includes text samples and generated labels. The model parameters include: weight parameters, momentum parameters, learning rate, weight decay rate and decay parameters.

[0169] Text prediction module 720: It is used to input text samples into the initial large language model for text prediction, and use the model parameters for forward propagation to obtain the predicted text of the text samples.

[0170] Parameter update module 730: It is used to obtain the loss function based on the predicted text and generated labels, calculate the gradient value of the loss function with respect to the model parameters, update the momentum parameters using the decay parameters and gradient values, and update the weight parameters using the learning rate, weight decay rate and updated momentum parameters. The weight parameters are iterated multiple times until the initial large language model is trained and the target large language model is obtained.

[0171] Text generation module 740: Used to acquire the text to be processed and input the text to be processed into the target large language model to obtain the target generated text.

[0172] The specific implementation of the text generation device in this embodiment is basically the same as the specific implementation of the text generation method described above, and will not be repeated here.

[0173] This application also provides an electronic device, including:

[0174] At least one memory;

[0175] At least one processor;

[0176] At least one program;

[0177] The program is stored in a memory, and the processor executes the at least one program to implement the text generation method described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0178] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0179] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0180] The memory 802 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the text generation method of the embodiments of this application.

[0181] The 803 input / output interface is used to implement information input and output.

[0182] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0183] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0184] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0185] This application embodiment also provides a storage medium that stores a computer program, which, when executed by a processor, implements the above-described text generation method.

[0186] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] The text generation method, apparatus, device, and storage medium proposed in this application obtain a training text set and model parameters of an initial large language model. Text samples are input into the initial large language model for text prediction. Forward propagation is performed using the model parameters to obtain the predicted text of the text samples. A loss function is obtained based on the predicted text and generated labels, and the gradient value of the loss function with respect to the model parameters is calculated. Momentum parameters are updated using decay parameters and gradient values, and weight parameters are updated using the learning rate, weight decay rate, and the updated momentum parameters. The weight parameters are iterated multiple times until the initial large language model is trained, resulting in a target large language model. The text to be processed is obtained and input into the target large language model to obtain the target generated text. In this application embodiment, the large language model uses momentum parameters as intermediate state parameters during training. Compared to the AdamW optimizer in related technologies, this reduces intermediate state parameters, lowers memory usage, improves processor resource utilization, reduces the training cost of the large language model, and thus expands the applicability of the large language model.

[0188] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0189] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0190] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0192] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0193] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0195] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A text generation method, characterized in that, include: Obtain the training text set and the model parameters of the initial large language model. The training text set includes text samples and generated labels. The model parameters include: weight parameters, momentum parameters, learning rate, weight decay rate, and decay parameters. The text sample is input into the initial large language model for text prediction, and the model parameters are used for forward propagation to obtain the predicted text of the text sample. The loss function is obtained based on the predicted text and the generated labels, and the gradient value of the loss function with respect to the model parameters is calculated. The momentum parameter is updated using the decay parameter and the gradient value, and the weight parameter is updated using the learning rate, the weight decay rate and the updated momentum parameter. The weight parameter is iterated multiple times until the initial large language model is trained and the target large language model is obtained. Obtain the text to be processed and input it into the target large language model to obtain the target generated text.

2. The text generation method according to claim 1, characterized in that, The step of updating the momentum parameter using the decay parameter and the gradient value includes: The first value is calculated based on the decay parameter and the momentum parameter of the previous moment; The second value is calculated based on the attenuation parameter and the gradient value; The momentum parameter at the current moment is obtained based on the first value and the second value, and the momentum parameter at the current moment is used as the updated momentum parameter.

3. The text generation method according to claim 2, characterized in that, The step of updating the weight parameters using the learning rate, the weight decay rate, and the updated momentum parameters includes: The third value is calculated based on the weight parameters from the previous moment and the weight decay rate. The fourth value is calculated by using the third value and the momentum parameter at the current moment as inputs to the sign function; Calculate the fifth value based on the learning rate and the fourth value; The difference between the weight parameter at the previous moment and the fifth value is calculated to obtain the weight parameter at the current moment, and the weight parameter at the current moment is used as the updated weight parameter.

4. The text generation method according to claim 1, characterized in that, The weight parameters include the bias value of the linear layer, the scaling value of the normalized layer, the translation value of the normalized layer, and other parameters; the calculation process of the learning rate includes the following steps: If the weight parameter is the bias value, the scaling value, or the translation value, then the learning rate is obtained based on the global relative learning rate; If the weight parameter is one of the other parameters, the learning rate is obtained based on the root mean square of the weight parameter and the global relative learning rate.

5. The text generation method according to claim 4, characterized in that, The calculation process for the attenuation rate includes the following steps: If the weight parameter is the bias value, the scaling value, or the translation value, then the attenuation rate is set to zero. If the weight parameter is one of the other parameters, then the decay rate is set to a preset constant greater than zero.

6. The text generation method according to claim 3, characterized in that, The model parameters also include a gradient accumulation metric. Updating the momentum parameter using the decay parameter and the gradient value, and updating the weight parameter using the learning rate, the weight decay rate, and the updated momentum parameter, includes: A sixth value is calculated using the current momentum parameter, the decay parameter, and the gradient accumulation index, and a seventh value is calculated based on the gradient value, the decay parameter, and the number of weights in the weight parameter. The updated momentum parameter is obtained based on the sum of the sixth value and the seventh value. Multiply the gradient accumulation index and the fifth value to obtain the eighth value; The updated weight parameter is obtained by calculating the difference between the current weight parameter and the eighth value.

7. The text generation method according to claim 6, characterized in that, The calculation process of the gradient accumulation index includes: Get the iteration time of the current iteration; If the iteration time is divisible by the number of weights, the gradient accumulation index is a first preset value; otherwise, the gradient accumulation index is a second preset value.

8. A text generation device, characterized in that, include: Data acquisition module: used to acquire the training text set and the model parameters of the initial large language model. The training text set includes text samples and generated labels. The model parameters include: weight parameters, momentum parameters, learning rate, weight decay rate and decay parameters. Text prediction module: used to input the text sample into the initial large language model for text prediction, and use the model parameters for forward propagation to obtain the predicted text of the text sample; Parameter update module: used to obtain a loss function based on the predicted text and the generated labels, calculate the gradient value of the loss function with respect to the model parameters, update the momentum parameter using the decay parameter and the gradient value, update the weight parameter using the learning rate, the weight decay rate and the updated momentum parameter, iterate the weight parameter multiple times until the initial large language model is trained and the target large language model is obtained; Text generation module: used to acquire the text to be processed and input the text to be processed into the target large language model to obtain the target generated text.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the text generation method according to any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the text generation method according to any one of claims 1 to 7.

Images