Model training method, text classification method, device, electronic device and medium

Abstract

The embodiments of the present application provide a model training method, a text classification method, an apparatus, an electronic device, and a medium, which belong to the field of artificial intelligence technology. The method includes: in the current iteration round, adding a random seed and a projection parameter to a seed queue and a projection queue respectively, obtaining a target random seed from the seed queue in turn, generating a target Gaussian pseudo-random vector according to the target random seed, obtaining the projection parameter in the projection queue according to the target random seed, obtaining the target projection parameter, and multiplying the target Gaussian pseudo-random vector and the target projection parameter to determine the target zero-order gradient estimate, and updating the model parameters of the preset large language model according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset number of iterations threshold, thereby obtaining the target large language model, which can reduce the memory space required for model training.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, a text classification method, a device, an electronic device, and a medium. Background Art

[0002] In related technologies, first-order optimization algorithms based on automatic differentiation are typically used to train large language models. Specifically, given a training sample and a desired target, the model's output under the given training sample is calculated as the derivative of the loss value with respect to the desired target with respect to the model parameters. The model gradient is estimated, and after multiple iterations, the optimal model parameters are gradually approached. However, this training method typically consumes a large amount of memory. For example, the OPT series of large language models typically consumes 7 to 12 times more memory for training calculations than for inference calculations. Therefore, reducing the memory space required for model training has become a pressing issue. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method, text classification method, device, electronic device and medium, aiming to reduce the memory space required for model training.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model training method, the method comprising:

[0005] Initialize the seed queue and projection queue;

[0006] In the current iteration round, a random seed, a natural language text, and a text category label of the natural language text are obtained;

[0007] Determining projection parameters of a preset large language model according to the random seed, the natural language text, and the text category label;

[0008] Adding the random seed to the seed queue, and adding the projection parameter to the projection queue;

[0009] Obtaining a target random seed from the seed queue in sequence, generating a target Gaussian pseudo-random vector according to the target random seed, obtaining projection parameters in the projection queue according to the target random seed, obtaining target projection parameters, and multiplying the target Gaussian pseudo-random vector and the target projection parameters to determine a target zero-order gradient estimate;

[0010] For each iteration round, the model parameters of the preset large language model are updated according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model; wherein, the target large language model is used to perform text classification on the input target text.

[0011] In some embodiments, determining projection parameters of a preset large language model based on the random seed, the natural language text, and the text category label includes:

[0012] generating an initial Gaussian pseudo-random vector according to the random seed;

[0013] Performing forward perturbation on model parameters of the preset large language model according to the initial Gaussian pseudo-random vector and a preset perturbation step size to obtain forward perturbation parameters;

[0014] Determining a first loss based on the forward perturbation parameter, the natural language text, and the text category label;

[0015] Performing backward perturbation on the forward perturbation parameter according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain a backward perturbation parameter;

[0016] Determine projection parameters of a preset large language model according to the first loss, the backward perturbation parameter, the natural language text, and the text category label.

[0017] In some embodiments, determining the projection parameters of a preset large language model based on the first loss, the backward perturbation parameter, the natural language text, and the text category label includes:

[0018] Determining a second loss based on the backward perturbation parameter, the natural language text, and the text category label;

[0019] Determine projection parameters of a preset large language model according to the first loss, the second loss, and the preset perturbation step size.

[0020] In some embodiments, adding the random seed to the seed queue and adding the projection parameter to the projection queue includes:

[0021] Obtaining the queue length of the seed queue; both the seed queue and the projection queue follow the first-in-first-out principle;

[0022] If the queue length is less than a preset length threshold, adding the random seed to the seed queue and adding the projection parameter to the projection queue;

[0023] If the queue length is equal to the preset length threshold, the earliest random seed is popped out from the seed queue and the earliest projection parameter is popped out from the projection queue according to the first-in-first-out principle, and the random seed is added to the seed queue and the projection parameter is added to the projection queue.

[0024] In some embodiments, updating the model parameters of the preset large language model according to a preset learning rate, a preset momentum parameter, a position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches a preset iteration number threshold to obtain a target large language model includes:

[0025] Determine momentum decision parameters;

[0026] If the momentum decision parameter indicates that the model parameters are updated using a momentum variable, then preliminarily updating the model parameters of the preset large language model according to the preset learning rate, the random seed, and the projection parameter;

[0027] The model parameters after the preliminary update are updated again according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model.

[0028] In some embodiments, the updating of the model parameters after the initial update according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate includes:

[0029] Determining an exponent of the preset momentum parameter according to a position identifier of the target random seed in the seed queue, performing exponential calculation on the position identifier and the preset momentum parameter to obtain an initial update parameter;

[0030] Multiplying the preset learning rate, the initial update variable, and the target zero-order gradient estimate to obtain a reference update parameter;

[0031] The model parameters after the initial update are updated again according to the reference update parameters.

[0032] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application proposes a text classification method, which includes:

[0033] Get the target text;

[0034] The target text is classified by a target large language model to obtain a target text category; wherein the target large language model is trained according to the model training method described in the first aspect.

[0035] To achieve the above-mentioned purpose, a third aspect of the embodiments of the present application provides a text classification device, comprising:

[0036] Acquisition module, used to obtain target text;

[0037] A classification module is used to perform text classification on the target text through a target large language model to obtain a target text category; wherein, the target large language model is trained according to the model training method described in the first aspect.

[0038] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the model training method described in the first aspect or the text classification method described in the second aspect.

[0039] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the model training method described in the first aspect or the text classification method described in the second aspect.

[0040] The model training method, text classification method, text classification device, electronic device, and computer-readable storage medium of the embodiments of the present application initialize a seed queue and a projection queue to store random seeds and projection parameters based on the seed queue and projection queue, respectively. In the current iteration, a random seed, natural language text, and a text category label for the natural language text are obtained, and a model training task is performed based on the random seed, natural language text, and text category label. Based on the random seed, natural language text, and text category label, projection parameters of a preset large language model are determined. Zero-order gradient estimation is performed on data pairs consisting of the random seed and projection parameters. The model gradient is determined by the random seed and projection parameters. By only storing the random seed and projection parameters, the model gradient can be determined, avoiding the additional memory overhead required to record historical model gradients using a first-order optimization algorithm, thereby reducing the memory space required for model training. The random seed is added to the seed queue, and the projection parameters are added to the projection queue. Model optimization is performed based on the random seed in the seed queue and the projection parameters in the projection queue. A target random seed is sequentially obtained from the seed queue, and a target Gaussian pseudo-random vector is generated based on the target random seed. Saving the random seed is equivalent to saving the Gaussian pseudo-random vector, reducing memory overhead. The projection parameters in the projection queue are obtained according to the target random seed to obtain the target projection parameters, and the target Gaussian pseudo-random vector and the target projection parameters are multiplied to perform gradient estimation based on the seed-projection pair to determine the target zero-order gradient estimate. For each iteration round, the model parameters of the preset large language model are updated according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate, so that the model training process only needs to pass through the random seeds in the seed queue and the projection parameters in the projection queue, without the need to open up additional storage space other than the seed queue and the projection queue. The large language model with the best performance is determined through multiple iterations until the current iteration round reaches the preset iteration number threshold, and the target large language model is obtained. The entire model training process can be carried out through the seed-projection pair, and there is no need to open up additional memory space to record historical gradient information, thereby reducing the memory space required for model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the model training method provided in the embodiment of the present application;

[0042] Figure 2 yes Figure 1 Flowchart of step S130 in FIG.

[0043] Figure 3 yes Figure 2 Flowchart of step S250 in FIG.

[0044] Figure 4 yes Figure 1Flowchart of step S140 in FIG.

[0045] Figure 5 yes Figure 1 Flowchart of step S160 in FIG.

[0046] Figure 6 yes Figure 5 Flowchart of step S530 in FIG.

[0047] Figure 7 is a flowchart of the text classification method provided in an embodiment of the present application;

[0048] Figure 8 is a structural diagram of a text classification device provided in an embodiment of the present application;

[0049] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0053] In related technologies, first-order optimization algorithms based on automatic differentiation are typically used to train large language models. Specifically, given a training sample and a desired target, the model's output under the given training sample is calculated as the derivative of the loss value with respect to the desired target with respect to the model parameters. The model gradient is estimated, and after multiple iterations, the optimal model parameters are gradually approached. However, this training method typically consumes a large amount of memory. For example, the OPT series of large language models typically consumes 7 to 12 times more memory for training calculations than for inference calculations. Therefore, reducing the memory space required for model training has become a pressing issue.

[0054] Based on this, embodiments of the present application provide a model training method, a text classification method, a text classification device, an electronic device, and a computer-readable storage medium, aiming to reduce the memory space required for model training.

[0055] The model training method, text classification method, text classification device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the model training method in the embodiments of the present application is described.

[0056] The model training method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The model training method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the model training method, etc., but is not limited to the above forms.

[0057] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0058] Figure 1 This is an optional flowchart of the model training method provided in an embodiment of the present application. The model training method may include but is not limited to steps S110 to S160.

[0059] Step S110, initializing the seed queue and projection queue;

[0060] Step S120: In the current iteration round, a random seed, a natural language text, and a text category label of the natural language text are obtained;

[0061] Step S130, determining projection parameters of a preset large language model based on the random seed, the natural language text, and the text category label;

[0062] Step S140, adding the random seed to the seed queue and the projection parameters to the projection queue;

[0063] Step S150, sequentially obtaining a target random seed from the seed queue, generating a target Gaussian pseudo-random vector according to the target random seed, obtaining a projection parameter in the projection queue according to the target random seed, obtaining the target projection parameter, and multiplying the target Gaussian pseudo-random vector and the target projection parameter to determine a target zero-order gradient estimate;

[0064] Step S160: For each iteration round, the model parameters of the preset large language model are updated according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate, until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model; wherein the target large language model is used to perform text classification on the input target text.

[0065] In step S110 of some embodiments, two queues are initialized, serving as a seed queue and a projection queue, respectively. A queue is a data structure that follows the first-in-first-out principle, a data processing principle in which the earliest data entering the queue is processed or removed from the queue first. Both the seed queue and the projection queue have a maximum length limit of h. The seed queue can accommodate a maximum of h random seeds, and the projection queue can accommodate a maximum of h projection parameters. Each random seed is associated with a corresponding projection parameter, and a gradient estimate can be determined based on the seed-projection pair formed by the random seed and the projection parameter.

[0066] In step S120 of some embodiments, the pre-training or fine-tuning process of the centralized large language model needs to go through multiple iterations, and each iteration needs training samples to support the training process. In the current iteration, a random integer is initialized or a random integer is pre-set to obtain a random seed, and natural language text and text category labels of natural language text are obtained from the training data set as training samples. Natural language text is text in natural language form and conforms to specific grammatical rules. It can be a book, an article, a conversation text, a social media post, etc. The text category label is the label used when classifying natural language text. The text category label can be set based on different text classification tasks. If the text classification task is intent recognition, the text category label is the intent category, such as query, purchase, consultation, etc. If the text classification task is emotion recognition, the text category label is the emotion category, such as positive, negative or neutral.

[0067] The embodiment of the present application determines the zero-order gradient estimate required for model training based on the seed-projection pair, so it is necessary to obtain projection parameters that match the random seed.

[0068] See also Figure 2 In some embodiments, step S130 may include but is not limited to steps S210 to S250:

[0069] Step S210, generating an initial Gaussian pseudo-random vector according to a random seed;

[0070] Step S220, performing forward perturbation on the model parameters of the preset large language model according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain forward perturbation parameters;

[0071] Step S230, determining a first loss based on the forward perturbation parameter, the natural language text, and the text category label;

[0072] Step S240, performing backward perturbation on the forward perturbation parameter according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain the backward perturbation parameter;

[0073] Step S250: Determine projection parameters of a preset large language model based on the first loss, the backward perturbation parameter, the natural language text, and the text category label.

[0074] In step S210 of some embodiments, a random seed is calculated using a pseudo-random number generation algorithm to obtain an initial Gaussian pseudo-random vector. The pseudo-random number generation algorithm can be a Philox algorithm, a linear feedback shift register (LFSR) algorithm, a linear congruential generator (LCG) algorithm, etc. The initial Gaussian pseudo-random vector is a vector of very large length.

[0075] In step S220 of some embodiments, in order to reduce the memory space occupied by model training to the inference level, the embodiment of the present application adopts a zero-order optimization method for model training. The zero-order optimization method can avoid the complex calculation of back propagation, thereby significantly reducing the memory requirement of model training, which is especially important for the training of large language models. The preset perturbation step size is a hyperparameter used to control the perturbation amplitude of the model parameters. The initial Gaussian pseudo-random vector and the preset perturbation step size are multiplied to determine the random perturbation. In-place model modification is performed on the memory occupied by the preset large language model, and a random perturbation of one times the forward direction is applied in-place to the model parameters of the preset large language model to obtain new model parameters and obtain forward perturbation parameters. The preset large language model refers to a deep learning model trained using a large amount of text data, such as a GPT series model. In the current iteration round, if the model parameters of the preset large language model are w, the preset perturbation step size is μ, and the initial Gaussian pseudo-random vector is z, then the forward perturbation parameter is expressed as w+μz.

[0076] In some embodiments, in step S230, after the model parameters of the preset large language model are updated in situ as forward perturbation parameters, a first large language model can be determined. Natural language text is input into the first large language model, and the text category of the natural language text is predicted to obtain a first text category. Based on a loss function, a loss is calculated based on the text category label and the first text category to obtain a first loss. The loss function can be a cross-entropy loss function.

[0077] In step S240 of some embodiments, referring to step S220, the initial Gaussian pseudo-random vector and the preset disturbance step size are multiplied to determine the random disturbance. A random disturbance twice as much as the backward disturbance is applied to the forward disturbance parameter to obtain the backward disturbance parameter. In the current iteration round, if the model parameter of the preset large language model is w, the preset disturbance step size is μ, and the initial Gaussian pseudo-random vector is z, then the backward disturbance parameter can be expressed as w-μz. If the forward disturbance parameter is w1, then the backward disturbance parameter can be expressed as w1-2*μz, where * represents a multiplication operation. It should be noted that due to the introduction of the pseudo-random number generation algorithm, saving the random seed is equivalent to saving the Gaussian pseudo-random vector, so that the model disturbance can be performed in situ without the need to open up a new memory to store the models before and after the disturbance, so as to compress the memory space required for zero-order gradient estimation. This allows some devices with smaller algorithms, such as mobile phones and tablets, to train some larger models under specific conditions.

[0078] A random perturbation twice as large as the forward perturbation is applied to the backward perturbation parameter in situ, and the backward perturbation parameter is restored in situ to the initial model parameter w of the current iteration round.

[0079] In step S250 of some embodiments, the loss change of the loss function between the forward perturbation parameter and the backward perturbation parameter is determined based on the first loss, the backward perturbation parameter, the natural language text and the text category label to obtain the projection parameter.

[0080] In the above steps S210 to S250, forward perturbations and backward perturbations are applied to the model to determine the projection parameters, and the zero-order gradient estimation is calculated based on the data pairs consisting of the random seed and the projection parameters. Compared with the first-order optimization algorithm that needs to record historical gradient information, the memory overhead of model training is reduced.

[0081] See also Figure 3 In some embodiments, step S250 may include but is not limited to steps S310 to S320:

[0082] Step S310, determining a second loss based on the backward perturbation parameter, the natural language text, and the text category label;

[0083] Step S320 : Determine projection parameters of a preset large language model according to the first loss, the second loss, and the preset perturbation step size.

[0084] In some embodiments, in step S310, after the forward perturbation parameters are updated in situ to the backward perturbation parameters, a second language model may be determined. Natural language text is input into the second language model to predict the text category of the natural language text, thereby obtaining a second text category. Based on a loss function, a loss is calculated based on the text category label and the second text category to obtain a second loss. The loss function may employ a cross-entropy loss function.

[0085] In step S320 of some embodiments, the first loss is subtracted from the second loss to obtain a loss change. The ratio between the loss change and 2 times the preset perturbation step size is calculated to obtain a projection parameter. The calculation formula of the projection parameter is defined as:

[0086] ,

[0087] Where p is the projection parameter; For the first loss; For the second loss; is the preset perturbation step size.

[0088] Through the above steps S310 to S320, projection parameters can be obtained, so as to determine gradient estimation based on the projection parameters to perform model update.

[0089] See also Figure 4 In some embodiments, step S140 may include but is not limited to steps S410 to S430:

[0090] Step S410: Obtain the queue length of the seed queue; both the seed queue and the projection queue follow the first-in-first-out principle;

[0091] Step S420: If the queue length is less than the preset length threshold, the random seed is added to the seed queue and the projection parameter is added to the projection queue;

[0092] Step S430: If the queue length is equal to the preset length threshold, the earliest random seed is popped out from the seed queue and the earliest projection parameter is popped out from the projection queue according to the first-in-first-out principle, and the random seed is added to the seed queue and the projection parameter is added to the projection queue.

[0093] In step S410 of some embodiments, the queue length of the seed queue is obtained, where the queue length is the number of random seeds stored in the seed queue.

[0094] In step S420 of some embodiments, the preset length threshold is the maximum length limit h of the seed queue. If the queue length is less than the preset length threshold, it indicates that the seed queue has remaining storage space for storing new random seeds, and the random seed is added to the seed queue. Since the random seed and projection parameters are paired, the projection parameters can be added to the projection queue.

[0095] In step S430 of some embodiments, if the queue length is equal to a preset length threshold, indicating that both the seed queue and the projection queue are full and have no remaining storage space to store new random seeds and new projection parameters, the oldest random seed is removed from the seed queue and the oldest projection parameter is removed from the projection queue according to a first-in-first-out principle, thereby ignoring random seeds and projection parameters that are too old, and thus ignoring historical gradient information that is too old. After the oldest random seed and projection parameter are removed from the seed queue and the projection queue, the seed queue and the projection queue have excess storage space for storing new random seeds and new projection parameters, and the random seed can be added to the seed queue, and the projection parameter can be added to the projection queue.

[0096] In the above steps S410 to S430, by limiting the queue length of the seed queue, random seeds and projection parameters that are too old can be ignored, thereby saving the storage space required for model training.

[0097] In step S150 of some embodiments, in the current iteration round, a random seed is obtained from the seed queue in sequence according to the first-in-first-out principle to obtain a target random seed, the target random seed is calculated by a pseudo-random number generation algorithm to generate a target Gaussian pseudo-random vector, projection parameters matching the target random seed are obtained from the projection queue to obtain target projection parameters, and the target Gaussian pseudo-random vector and the target projection parameters are multiplied to determine the target zero-order gradient estimate.

[0098] The first-order optimization algorithm will record historical gradient information, that is, in addition to the memory space required by the automatic differentiation algorithm, additional memory overhead is required. By exchanging memory for time, the model convergence is accelerated, which makes it impossible to train the model under specific conditions with only double the inference memory overhead. For example, the model update method of momentumSGD is w←w-η(g+αb), where w is the model weight, η is the learning rate, g is the gradient estimate, α is the momentum parameter, the momentum parameter is greater than 0 and less than 1, and b is the momentum, which is a vector with the same length as g. Momentum is actually a linear combination of historical gradient information. The vector b is calculated by g many times before, so additional memory space needs to be opened up to store the gradient estimates g many times before. The embodiment of the present application limits the queue length of the seed queue and the projection queue, ignores historical gradient information that is too old, and uses the seed-projection pair of the seed queue and the projection queue to represent the historical gradient information of a limited number of steps, thereby reducing the storage space required for historical gradient information to an extremely low level.

[0099] See also Figure 5 In some embodiments, step S160 may include but is not limited to steps S510 to S530:

[0100] Step S510, determining momentum decision parameters;

[0101] Step S520: If the momentum decision parameter indicates that the momentum variable is used to update the model parameters, the model parameters of the preset large language model are preliminarily updated according to the preset learning rate, random seed, and projection parameter;

[0102] In step S530, the model parameters after the initial update are updated again according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model.

[0103] In step S510 of some embodiments, a momentum decision parameter is obtained. The momentum decision parameter is used to indicate whether Nesterov momentum is used in the current iteration round of model training. The momentum decision parameter is a Boolean variable whose value is 0 or 1. If the momentum decision parameter is 1, Nesterov momentum is used in the current iteration round of model training; otherwise, Nesterov momentum is not used.

[0104] In some embodiments, in step S520, if the momentum decision parameter indicates that the model parameters should be updated using a momentum variable, the momentum decision parameter is 1. An initial Gaussian pseudo-random vector is generated based on the random seed, and the initial Gaussian pseudo-random vector is multiplied by the projection parameter to obtain an initial zero-order gradient estimate. The model parameters of the preset large language model are initially updated based on a preset learning rate and the initial zero-order gradient estimate. The preset learning rate is used to control the step size of the model parameter update during model training. The formula for the initial update is defined as: w←w–ηg, where w is the model parameter of the preset large language model, η is the preset learning rate, and g is the initial zero-order gradient estimate. If the momentum decision parameter indicates that the model parameters should not be updated using a momentum variable, step S520 is skipped and, referring to step S530, the model parameters of the preset large language model are updated based on the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches a preset iteration number threshold, thereby obtaining the target large language model.

[0105] In step S530 of some embodiments, the preset momentum parameter is a hyperparameter for accelerating the gradient descent process, and the preset momentum parameter is greater than 0 and less than 1. The seed queue is represented as slist. Assuming that the seed queue contains h random seeds, the jth random seed is obtained from the seed queue according to the first-in-first-out principle as the target random seed slist[j]. The model parameters that have been initially updated in the current iteration round are updated again according to the preset learning rate, the preset momentum parameter, the position identifier j of the target random seed slist[j] in the seed queue, and the target zero-order gradient estimate corresponding to the target random seed slist[j], to obtain the intermediate model parameters updated based on the current target random seed. Continue to obtain the next target random seed slist[j+1] from the seed queue, and estimate the target zero-order gradient corresponding to the target random seed slist[j+1] according to the preset learning rate, preset momentum parameter, the position j+1 of the target random seed slist[j+1] in the seed queue, and the target random seed slist[j+1]. Update and output intermediate model parameters , until the intermediate model parameters based on the last target random seed output in the seed queue are obtained , completing the update of the model parameters in the current iteration round, and repeating the iterative update process from step S120 to step S160 until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model. It should be noted that each iteration round updates the model parameters of the preset large language model updated in the previous iteration round.

[0106] In steps S510 to S530, the momentum decision parameter is used to control the model training process, accelerating model convergence and improving model training efficiency. Furthermore, by iteratively updating the model parameters using the random seeds stored in the seed queue, the model parameters can be updated, ignoring historical gradient information that is too old and eliminating the need to store historical gradient information. This saves memory space required for model training compared to first-order algorithms.

[0107] See also Figure 6 In some embodiments, step S530 may include but is not limited to steps S610 to S650:

[0108] Step S610, determining an exponent of a preset momentum parameter according to a position identifier of the target random seed in the seed queue, performing exponential calculation on the position identifier and the preset momentum parameter to obtain an initial update parameter;

[0109] Step S620, multiplying the preset learning rate, the initial update variable, and the target zero-order gradient estimate to obtain a reference update parameter;

[0110] Step S630: Update the initially updated model parameters again according to the reference update parameters.

[0111] In step S610 of some embodiments, a dequeue identifier is determined based on the queue length of the seed queue and the position identifier of the target random seed in the seed queue. If the position identifier is j and the queue length is h, the dequeue identifier is h-j+1. The seed queue follows the first-in-first-out principle. To facilitate the entry and exit of random seeds, the most recently added random seeds have a smaller position identifier in the seed queue and a larger dequeue identifier. The random seed added to the seed queue in the current iteration round has a position identifier of 1 and a dequeue identifier of h.

[0112] The dequeue flag is used as the exponent of the preset momentum parameter, and the exponential calculation of the dequeue flag and the preset momentum parameter is performed to obtain the initial update parameter. The initial update parameter is expressed as:

[0113] ,

[0114] in, is the preset momentum parameter, which is greater than 0 and less than 1; h is the queue length of the seed queue; j is the position identifier of the target random seed in the seed queue.

[0115] In step S620 of some embodiments, a preset learning rate, an initial update variable, and a target zero-order gradient estimate are multiplied to obtain a reference update parameter.

[0116] In step S630 of some embodiments, the model parameters after the initial update are subtracted from the reference update parameters to update the model parameters after the initial update. The formula for the further update is defined as:

[0117] ,

[0118] Among them, w is the model parameter after preliminary update; is the preset learning rate; is the preset momentum parameter; h is the queue length; p is the target projection parameter; z is the target Gaussian pseudo-random vector.

[0119] Momentum b is a linear combination of historical gradient information. The momentum of the embodiment of the present application can be expressed as b=g+α*g_1+α^2*g_2+…+α^k*g_k+…+α^h*g_h, where g_k represents the gradient information recorded when the gradient descends k steps away from the current step, ^ represents exponential operation, and * represents multiplication operation. Note that 0<α<1, and the influence of gradient information that is too old on momentum b will actually gradually decrease. Therefore, when the historical window length h is not too small, the historical gradient reconstructed by the seed queue and projection queue will not have a large error with the method of explicitly recording momentum. At the same time, when using the zero-order optimization algorithm to update the model parameters, it is no longer necessary to open up additional memory space to record momentum. Instead, it is only necessary to record h matching seeds and projections, which greatly reduces the memory overhead of model training.

[0120] Through the above steps S610 to S630, the influence of historical gradient information that is too old on model training is reduced, and the historical gradient can be determined by recording the random seed and projection parameters without the need to allocate additional storage space to store the historical gradient. While ensuring the model classification performance, the memory space required for model training is reduced.

[0121] Figure 7 This is a flowchart of the text classification method provided by an embodiment of the present application, which may include but is not limited to steps S710 to S720:

[0122] Step S710, obtaining target text;

[0123] Step S720 , classifying the target text using the target large language model to obtain a target text category.

[0124] In step S710 of some embodiments, a target text is obtained, where the target text is the text to be classified.

[0125] In step S720 of some embodiments, the target large language model trained by the above-mentioned model training method performs text classification on the target text and outputs the target text category.

[0126] In the above steps S710 to S720, the text category of the target text is predicted by the large language model, which can accurately understand the text semantics of the target text and improve the accuracy of text classification.

[0127] See also Figure 8 The present application also provides a text classification device that can implement the above-mentioned text classification method. The text classification device includes:

[0128] An acquisition module 810 is used to acquire a target text;

[0129] The classification module 820 is used to classify the target text using the target large language model to obtain the target text category.

[0130] The specific implementation of the text classification device is basically the same as the specific embodiment of the above-mentioned text classification method, and will not be repeated here.

[0131] The present application also provides a model training device that can implement the above-mentioned model training method. The model training device includes:

[0132] Initialization module, used to initialize the seed queue and projection queue;

[0133] A first acquisition module is used to obtain a random seed, a natural language text, and a text category label of the natural language text in a current iteration round;

[0134] A determination module, configured to determine projection parameters of a preset large language model based on a random seed, natural language text, and text category labels;

[0135] Add modules for adding random seeds to the seed queue and projection parameters to the projection queue;

[0136] a second acquisition module, configured to sequentially acquire a target random seed from the seed queue, generate a target Gaussian pseudo-random vector according to the target random seed, acquire projection parameters from the projection queue according to the target random seed, obtain target projection parameters, and multiply the target Gaussian pseudo-random vector and the target projection parameters to determine a target zero-order gradient estimate;

[0137] An update module is configured to update the model parameters of a preset large language model for each iteration round according to a preset learning rate, a preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches a preset iteration number threshold, thereby obtaining a target large language model; wherein the target large language model is used to perform text classification on the input target text.

[0138] In some embodiments, the determining module is further configured to:

[0139] Generate an initial Gaussian pseudo-random vector based on a random seed;

[0140] Performing forward perturbation on model parameters of a preset large language model according to an initial Gaussian pseudo-random vector and a preset perturbation step size to obtain forward perturbation parameters;

[0141] Determine the first loss based on the forward perturbation parameters, the natural language text, and the text category label;

[0142] Performing backward perturbation on the forward perturbation parameter according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain the backward perturbation parameter;

[0143] Determine projection parameters of a preset large language model based on the first loss, the backward perturbation parameter, the natural language text, and the text category label.

[0144] In some embodiments, the determining module is further configured to:

[0145] Determine the second loss based on the backward perturbation parameters, the natural language text, and the text category label;

[0146] Determine projection parameters of a preset large language model according to the first loss, the second loss, and a preset perturbation step size.

[0147] In some embodiments, the adding module is further configured to:

[0148] Get the queue length of the seed queue; both the seed queue and the projection queue follow the first-in-first-out principle;

[0149] If the queue length is less than the preset length threshold, the random seed is added to the seed queue and the projection parameters are added to the projection queue;

[0150] If the queue length is equal to the preset length threshold, the earliest random seed is popped out from the seed queue and the earliest projection parameter is popped out from the projection queue according to the first-in-first-out principle, and the random seed is added to the seed queue and the projection parameter is added to the projection queue.

[0151] In some embodiments, the update module is further configured to:

[0152] Determine momentum decision parameters;

[0153] If the momentum decision parameter indicates that the momentum variable is used to update the model parameters, the model parameters of the preset large language model are preliminarily updated according to the preset learning rate, random seed and projection parameters;

[0154] The model parameters after the initial update are updated again according to the preset learning rate, preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold to obtain the target large language model.

[0155] In some embodiments, the update module is further configured to:

[0156] Determine the exponent of the preset momentum parameter according to the position identifier of the target random seed in the seed queue, perform exponential calculation on the position identifier and the preset momentum parameter, and obtain the initial update parameter;

[0157] Multiply the preset learning rate, the initial update variable, and the target zero-order gradient estimate to obtain the reference update parameter;

[0158] The model parameters after the initial update are updated again according to the reference update parameters.

[0159] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described model training method or text classification method. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0160] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0161] The processor 910 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0162] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called by the processor 910 to execute the model training method or text classification method of the embodiments of this application.

[0163] Input / output interface 930, used to implement information input and output;

[0164] Communication interface 940, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0165] bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0166] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0167] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned model training method or text classification method.

[0168] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0170] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0172] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0173] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0174] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0179] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A model training method, characterized in that: The method comprises: Initialize the seed queue and projection queue; In the current iteration round, a random seed, a natural language text, and a text category label of the natural language text are obtained; Determining projection parameters of a preset large language model according to the random seed, the natural language text, and the text category label; Adding the random seed to the seed queue, and adding the projection parameter to the projection queue; Obtaining a target random seed from the seed queue in sequence, generating a target Gaussian pseudo-random vector according to the target random seed, obtaining projection parameters in the projection queue according to the target random seed, obtaining target projection parameters, and multiplying the target Gaussian pseudo-random vector and the target projection parameters to determine a target zero-order gradient estimate; For each iteration round, the model parameters of the preset large language model are updated according to a preset learning rate, a preset momentum parameter, a position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches a preset iteration number threshold, thereby obtaining a target large language model; wherein the target large language model is used to perform text classification on the input target text; The updating of the model parameters of the preset large language model according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches a preset iteration number threshold to obtain the target large language model includes: Determine momentum decision parameters; If the momentum decision parameter indicates that the model parameters are updated using a momentum variable, then preliminarily updating the model parameters of the preset large language model according to the preset learning rate, the random seed, and the projection parameter; The model parameters after the preliminary update are updated again according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate, until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model; The updating of the model parameters after the preliminary update according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate includes: Determining an exponent of the preset momentum parameter according to a position identifier of the target random seed in the seed queue, performing exponential calculation on the position identifier and the preset momentum parameter to obtain an initial update parameter; Multiplying the preset learning rate, the initial update parameter, and the target zero-order gradient estimate to obtain a reference update parameter; The model parameters after the initial update are updated again according to the reference update parameters.

2. The method according to claim 1, characterized in that The step of determining projection parameters of a preset large language model according to the random seed, the natural language text, and the text category label includes: generating an initial Gaussian pseudo-random vector according to the random seed; Performing forward perturbation on model parameters of the preset large language model according to the initial Gaussian pseudo-random vector and a preset perturbation step size to obtain forward perturbation parameters; Determining a first loss based on the forward perturbation parameter, the natural language text, and the text category label; Performing backward perturbation on the forward perturbation parameter according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain a backward perturbation parameter; Determine projection parameters of a preset large language model according to the first loss, the backward perturbation parameter, the natural language text, and the text category label.

3. The method according to claim 2, characterized in that The determining of projection parameters of a preset large language model according to the first loss, the backward perturbation parameter, the natural language text, and the text category label includes: Determining a second loss based on the backward perturbation parameter, the natural language text, and the text category label; Determine projection parameters of a preset large language model according to the first loss, the second loss, and the preset perturbation step size.

4. The method according to claim 1, wherein The adding the random seed to the seed queue and the adding the projection parameter to the projection queue includes: Obtaining the queue length of the seed queue; both the seed queue and the projection queue follow the first-in-first-out principle; If the queue length is less than a preset length threshold, adding the random seed to the seed queue and adding the projection parameter to the projection queue; If the queue length is equal to the preset length threshold, the earliest random seed is popped out from the seed queue and the earliest projection parameter is popped out from the projection queue according to the first-in-first-out principle, and the random seed is added to the seed queue and the projection parameter is added to the projection queue.

5. A text classification method, characterized in that: The method comprises: Get the target text; The target text is classified by a target large language model to obtain a target text category; wherein the target large language model is trained according to the model training method according to any one of claims 1 to 4.

6. A text classification device, characterized in that: The device comprises: Acquisition module, used to obtain target text; A classification module is used to perform text classification on the target text through a target large language model to obtain a target text category; wherein, the target large language model is trained according to the model training method according to any one of claims 1 to 4.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the model training method according to any one of claims 1 to 4 or the text classification method according to claim 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model training method according to any one of claims 1 to 4 or the text classification method according to claim 5 is implemented.

Images