Model training method and device, text classification method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a model training method and device, a text classification method and device, electronic equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: in a current iteration round, adding a random seed and a projection parameter into a seed queue and a projection queue respectively, sequentially obtaining a target random seed from the seed queue, generating a target Gaussian pseudo-random vector according to the target random seed, obtaining the projection parameter in the projection queue according to the target random seed, and generating a target Gaussian pseudo-random vector according to the target Gaussian pseudo-random vector. And obtaining a target projection parameter, multiplying the target Gaussian pseudo-random vector by the target projection parameter, determining a target zero-order gradient estimation, and updating a model parameter of a preset large language model according to a preset learning rate, a preset momentum parameter, a position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimation. And the target large language model is obtained until the current iteration round reaches the preset iteration number threshold value, and the memory space needed by model training can be reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, a text classification method, a device, an electronic device and a medium. Background Art

[0002] In the related art, a first-order optimization algorithm based on automatic differentiation is usually used for model training of large language models. That is, given a training sample and an expected target, the derivative of the loss value between the model output and the expected target under the given training sample with respect to the model parameters is calculated, the model gradient is estimated, and the optimal model parameters are gradually approached after multiple iterations. However, this training method usually consumes a lot of memory. For example, the large language model OPT series usually takes up 7 to 12 times more memory for training calculations than inference calculations. Therefore, how to reduce the memory space required for model training has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method, a text classification method, a device, an electronic device and a medium, aiming to reduce the memory space required for model training.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model training method, the method comprising: Initialize the seed queue and projection queue; In the current iteration round, a random seed, a natural language text, and a text category label of the natural language text are obtained; Determining projection parameters of a preset large language model according to the random seed, the natural language text, and the text category label; Adding the random seed to the seed queue, and adding the projection parameter to the projection queue; Obtain a target random seed from the seed queue in sequence, generate a target Gaussian pseudo-random vector according to the target random seed, obtain projection parameters in the projection queue according to the target random seed, obtain target projection parameters, and multiply the target Gaussian pseudo-random vector and the target projection parameters to determine a target zero-order gradient estimate; For each iteration round, the model parameters of the preset large language model are updated according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model; wherein the target large language model is used to perform text classification on the input target text.

[0005] In some embodiments, determining the projection parameters of a preset large language model according to the random seed, the natural language text, and the text category label includes: Generate an initial Gaussian pseudo-random vector according to the random seed; Performing forward perturbation on model parameters of the preset large language model according to the initial Gaussian pseudo-random vector and a preset perturbation step size to obtain forward perturbation parameters; Determining a first loss according to the forward perturbation parameter, the natural language text, and the text category label; Performing backward perturbation on the forward perturbation parameter according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain a backward perturbation parameter; Determine projection parameters of a preset large language model according to the first loss, the backward perturbation parameter, the natural language text, and the text category label.

[0006] In some embodiments, determining the projection parameters of a preset large language model according to the first loss, the backward perturbation parameter, the natural language text, and the text category label includes: Determining a second loss according to the backward perturbation parameter, the natural language text, and the text category label; A projection parameter of a preset large language model is determined according to the first loss, the second loss, and the preset perturbation step size.

[0007] In some embodiments, adding the random seed to the seed queue and adding the projection parameter to the projection queue includes: Obtaining the queue length of the seed queue; both the seed queue and the projection queue follow the first-in-first-out principle; If the queue length is less than a preset length threshold, adding the random seed to the seed queue and adding the projection parameter to the projection queue; If the queue length is equal to the preset length threshold, the earliest random seed is popped out from the seed queue and the earliest projection parameter is popped out from the projection queue according to the first-in-first-out principle, and the random seed is added to the seed queue and the projection parameter is added to the projection queue.

[0008] In some embodiments, the updating of the model parameters of the preset large language model according to the preset learning rate, the preset momentum parameter, the position identifier of each of the target random seeds in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches a preset iteration number threshold, to obtain the target large language model, includes: Determine momentum decision parameters; If the momentum decision parameter indicates that the model parameters are updated using a momentum variable, the model parameters of the preset large language model are preliminarily updated according to the preset learning rate, the random seed and the projection parameter; The model parameters after the preliminary update are updated again according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model.

[0009] In some embodiments, the step of re-updating the initially updated model parameters according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate includes: Determine the exponent of the preset momentum parameter according to the position identifier of the target random seed in the seed queue, perform exponential calculation on the position identifier and the preset momentum parameter, and obtain an initial update parameter; Multiplying the preset learning rate, the initial update variable and the target zero-order gradient estimate to obtain a reference update parameter; The model parameters after the initial update are updated again according to the reference update parameters.

[0010] To achieve the above purpose, a second aspect of an embodiment of the present application proposes a text classification method, the method comprising: Get the target text; The target text is classified by a target large language model to obtain a target text category; wherein the target large language model is trained according to the model training method described in the first aspect.

[0011] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides a text classification device, the device comprising: An acquisition module is used to acquire the target text; A classification module is used to perform text classification on the target text through a target large language model to obtain a target text category; wherein the target large language model is trained according to the model training method described in the first aspect.

[0012] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method described in the first aspect or the text classification method described in the second aspect when executing the computer program.

[0013] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the model training method described in the first aspect or the text classification method described in the second aspect.

[0014] The model training method, text classification method, text classification device, electronic device and computer-readable storage medium of the embodiment of the present application are initialized by seed queue and projection queue to save random seeds and projection parameters based on seed queue and projection queue respectively. In the current iteration round, the random seed, natural language text and text category label of the natural language text are obtained to perform the model training task based on the random seed, natural language text and text category label. According to the random seed, natural language text and text category label, the projection parameters of the preset large language model are determined, and the zero-order gradient estimation is performed on the data pair composed of the random seed and the projection parameter. The model gradient is determined by the random seed and the projection parameter. The model gradient can be determined by only saving the random seed and the projection parameter, avoiding the need for additional memory overhead to record the historical model gradient using the first-order optimization algorithm, thereby reducing the memory space required for model training. The random seed is added to the seed queue, and the projection parameter is added to the projection queue to optimize the model based on the random seed in the seed queue and the projection parameter in the projection queue. A target random seed is obtained from the seed queue in turn, and a target Gaussian pseudo-random vector is generated according to the target random seed. Saving the random seed is equivalent to saving the Gaussian pseudo-random vector, thereby reducing the memory overhead. According to the target random seed, the projection parameters in the projection queue are obtained to obtain the target projection parameters, and the target Gaussian pseudo-random vector and the target projection parameters are multiplied to perform gradient estimation based on the seed-projection pair to determine the target zero-order gradient estimation. For each iteration round, the model parameters of the preset large language model are updated according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimation, so that the model training process only needs to pass the random seeds in the seed queue and the projection parameters in the projection queue, without opening up additional storage space other than the seed queue and the projection queue. The large language model with the best performance is determined through multiple iterations until the current iteration round reaches the preset iteration number threshold, and the target large language model is obtained. The entire model training process can be carried out through the seed-projection pair, and there is no need to open up additional memory space to record historical gradient information, thereby reducing the memory space required for model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the model training method provided in an embodiment of the present application; Figure 2 yes Figure 1 Flow chart of step S130 in FIG. Figure 3 yes Figure 2 Flow chart of step S250 in FIG. Figure 4 yes Figure 1 Flow chart of step S140 in FIG. Figure 5 yes Figure 1 Flow chart of step S160 in FIG. Figure 6 yes Figure 5 Flow chart of step S530 in FIG. Figure 7 is a flowchart of a text classification method provided in an embodiment of the present application; Figure 8 is a structural diagram of a text classification device provided in an embodiment of the present application; Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0019] In the related art, a first-order optimization algorithm based on automatic differentiation is usually used for model training of large language models. That is, given a training sample and an expected target, the derivative of the loss value between the model output and the expected target under the given training sample with respect to the model parameters is calculated, the model gradient is estimated, and the optimal model parameters are gradually approached after multiple iterations. However, this training method usually consumes a lot of memory. For example, the large language model OPT series usually takes up 7 to 12 times more memory for training calculations than inference calculations. Therefore, how to reduce the memory space required for model training has become an urgent problem to be solved.

[0020] Based on this, the embodiments of the present application provide a model training method, a text classification method, a text classification device, an electronic device and a computer-readable storage medium, aiming to reduce the memory space required for model training.

[0021] The model training method, text classification method, text classification device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the model training method in the embodiments of the present application is described.

[0022] The model training method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The model training method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the model training method, etc., but is not limited to the above forms.

[0023] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0024] Figure 1 It is an optional flowchart of the model training method provided in an embodiment of the present application. The model training method may include but is not limited to steps S110 to S160.

[0025] Step S110, initializing the seed queue and the projection queue; Step S120, in the current iteration round, obtaining a random seed, a natural language text, and a text category label of the natural language text; Step S130, determining projection parameters of a preset large language model according to the random seed, the natural language text, and the text category label; Step S140, adding the random seed to the seed queue and adding the projection parameter to the projection queue; Step S150, sequentially obtaining a target random seed from the seed queue, generating a target Gaussian pseudo-random vector according to the target random seed, obtaining a projection parameter in the projection queue according to the target random seed, obtaining a target projection parameter, and multiplying the target Gaussian pseudo-random vector and the target projection parameter to determine a target zero-order gradient estimate; Step S160, for each iteration round, the model parameters of the preset large language model are updated according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate, until the current iteration round reaches the preset iteration number threshold, and the target large language model is obtained; wherein the target large language model is used to perform text classification on the input target text.

[0026] In step S110 of some embodiments, two queues are initialized, and the two queues are used as a seed queue and a projection queue respectively. A queue is a data structure that follows the first-in-first-out principle, and the first-in-first-out principle is a data processing principle in which the data that enters the queue first is processed or removed from the queue first. The maximum length limit of the seed queue and the projection queue is h. The seed queue can accommodate up to h random seeds, and the projection queue can accommodate up to h projection parameters. Each random seed has a projection parameter that matches it, and the gradient estimate can be determined based on the seed-projection pair composed of the random seed and the projection parameter.

[0027] In step S120 of some embodiments, the pre-training or fine-tuning process of the centralized large language model needs to go through multiple iterations, and each iteration needs to have training samples that support the training process. In the current iteration, a random integer is initialized or a random integer is pre-set to obtain a random seed, and a natural language text and a text category label of the natural language text are obtained from the training data set as training samples. Natural language text is a text in the form of natural language and conforms to specific grammatical rules. It can be a book, an article, a conversation text, a social media post, etc. The text category label is a label used when classifying natural language text. The text category label can be set based on different text classification tasks. If the text classification task is intent recognition, the text category label is an intent category, such as query, purchase, consultation, etc. If the text classification task is emotion recognition, the text category label is an emotion category, such as positive, negative or neutral.

[0028] The embodiment of the present application determines the zero-order gradient estimate required for model training based on the seed-projection pair, and therefore needs to obtain projection parameters that match the random seed.

[0029] See also Figure 2 In some embodiments, step S130 may include but is not limited to steps S210 to S250: Step S210, generating an initial Gaussian pseudo-random vector according to a random seed; Step S220, forward perturb the model parameters of the preset large language model according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain forward perturbation parameters; Step S230, determining a first loss according to the forward perturbation parameter, the natural language text, and the text category label; Step S240, performing backward perturbation on the forward perturbation parameter according to the initial Gaussian pseudo-random vector and the preset perturbation step length to obtain the backward perturbation parameter; Step S250, determining projection parameters of a preset large language model according to the first loss, the backward perturbation parameter, the natural language text and the text category label.

[0030] In step S210 of some embodiments, the random seed is calculated by a pseudo-random number generation algorithm to obtain an initial Gaussian pseudo-random vector. The pseudo-random number generation algorithm can be a philox algorithm, a linear feedback shift register (LFSR) algorithm, a linear congruential generator (LCG) algorithm, etc. The initial Gaussian pseudo-random vector is a vector with a very large length.

[0031] In step S220 of some embodiments, in order to reduce the size of the memory space occupied by model training to the inference level, the embodiment of the present application adopts a zero-order optimization method for model training. The zero-order optimization method can avoid the complex calculation of back propagation, thereby significantly reducing the memory requirements of model training, which is particularly important for the training of large language models. The preset perturbation step size is a hyperparameter for controlling the perturbation amplitude of the model parameters. The initial Gaussian pseudo-random vector and the preset perturbation step size are multiplied to determine the random perturbation. In-place model modification is performed on the memory occupied by the preset large language model, and a random perturbation of one time forward is applied in-place to the model parameters of the preset large language model to obtain new model parameters and obtain forward perturbation parameters. The preset large language model refers to a deep learning model trained with a large amount of text data, such as a GPT series model. In the current iteration round, if the model parameters of the preset large language model are w, the preset perturbation step size is μ, and the initial Gaussian pseudo-random vector is z, the forward perturbation parameter is expressed as w+μz.

[0032] In step S230 of some embodiments, after the model parameters of the preset large language model are updated in situ as forward perturbation parameters, a first large language model can be determined. The natural language text is input into the first large language model, and the text category of the natural language text is predicted to obtain a first text category. Based on the loss function, the loss is calculated according to the text category label and the first text category to obtain a first loss. The loss function can adopt a cross entropy loss function.

[0033] In step S240 of some embodiments, referring to step S220, the initial Gaussian pseudo-random vector is multiplied by the preset disturbance step size to determine the random disturbance. A random disturbance twice as much as the backward disturbance is applied to the forward disturbance parameter to obtain the backward disturbance parameter. In the current iteration round, if the model parameter of the preset large language model is w, the preset disturbance step size is μ, and the initial Gaussian pseudo-random vector is z, the backward disturbance parameter can be expressed as w-μz. If the forward disturbance parameter is w1, the backward disturbance parameter can be expressed as w1-2*μz, where * represents a multiplication operation. It should be noted that due to the introduction of the pseudo-random number generation algorithm, saving the random seed is equivalent to saving the Gaussian pseudo-random vector, so that the model disturbance can be performed in situ without the need to open up a new memory to store the models before and after the disturbance respectively, so as to compress the memory space required for zero-order gradient estimation, which allows some devices with smaller algorithms such as mobile phones and tablets to train some larger models under specific conditions.

[0034] A random perturbation twice as large as the forward perturbation is applied to the backward perturbation parameters on the spot, and the backward perturbation parameters are restored on the spot to the initial model parameters w of the current iteration round.

[0035] In step S250 of some embodiments, the loss change of the loss function between the forward perturbation parameter and the backward perturbation parameter is determined based on the first loss, the backward perturbation parameter, the natural language text and the text category label to obtain the projection parameter.

[0036] In the above steps S210 to S250, forward perturbations and backward perturbations are applied to the model to determine the projection parameters, so as to calculate the zero-order gradient estimate based on the data pairs consisting of the random seed and the projection parameters. Compared with the first-order optimization algorithm that needs to record the historical gradient information, the memory overhead of model training is reduced.

[0037] See also Figure 3 In some embodiments, step S250 may include but is not limited to steps S310 to S320: Step S310, determining a second loss according to the backward perturbation parameter, the natural language text, and the text category label; Step S320: determining projection parameters of a preset large language model according to the first loss, the second loss and the preset perturbation step size.

[0038] In step S310 of some embodiments, after the forward perturbation parameters are updated in situ to the backward perturbation parameters, the second largest language model can be determined. The natural language text is input into the second largest language model, and the text category of the natural language text is predicted to obtain a second text category. Based on the loss function, the loss is calculated according to the text category label and the second text category to obtain a second loss. The loss function can adopt a cross entropy loss function.

[0039] In step S320 of some embodiments, the first loss is subtracted from the second loss to obtain a loss change. The ratio between the loss change and 2 times the preset disturbance step length is calculated to obtain a projection parameter. The calculation formula of the projection parameter is defined as: , Where p is the projection parameter; For the first loss; For the second loss; is the preset perturbation step size.

[0040] Through the above steps S310 to S320, projection parameters can be obtained, so as to determine the gradient estimation based on the projection parameters to perform model updating.

[0041] See also Figure 4 In some embodiments, step S140 may include but is not limited to steps S410 to S430: Step S410, obtaining the queue length of the seed queue; both the seed queue and the projection queue follow the first-in-first-out principle; Step S420, if the queue length is less than a preset length threshold, the random seed is added to the seed queue, and the projection parameter is added to the projection queue; Step S430: If the queue length is equal to the preset length threshold, the earliest random seed is popped out from the seed queue and the earliest projection parameter is popped out from the projection queue according to the first-in-first-out principle, and the random seed is added to the seed queue and the projection parameter is added to the projection queue.

[0042] In step S410 of some embodiments, the queue length of the seed queue is obtained, and the queue length is the number of random seeds stored in the seed queue.

[0043] In step S420 of some embodiments, the preset length threshold is the maximum length limit h of the seed queue. If the queue length is less than the preset length threshold, it means that there is still remaining storage space in the seed queue for storing new random seeds, and the random seeds are added to the seed queue. Since the random seeds and projection parameters are paired with each other, the projection parameters can be added to the projection queue.

[0044] In step S430 of some embodiments, if the queue length is equal to the preset length threshold, it means that both the seed queue and the projection queue are in a full data state, and there is no remaining storage space to store new random seeds and new projection parameters. Then, according to the first-in-first-out principle, the earliest random seed is popped out from the seed queue, and the earliest projection parameter is popped out from the projection queue to ignore the random seeds and projection parameters that are too old, thereby ignoring the historical gradient information that is too old. After the earliest random seeds and projection parameters are removed from the seed queue and the projection queue, the seed queue and the projection queue have excess storage space for storing new random seeds and new projection parameters, and the random seeds can be added to the seed queue, and the projection parameters can be added to the projection queue.

[0045] In the above steps S410 to S430, by limiting the queue length of the seed queue, random seeds and projection parameters that are too old can be ignored, thereby saving the storage space required for model training.

[0046] In step S150 of some embodiments, in the current iteration round, a random seed is obtained from the seed queue in sequence according to the first-in-first-out principle to obtain a target random seed, the target random seed is calculated by a pseudo-random number generation algorithm to generate a target Gaussian pseudo-random vector, projection parameters matching the target random seed are obtained from the projection queue to obtain target projection parameters, and the target Gaussian pseudo-random vector and the target projection parameters are multiplied to determine the target zero-order gradient estimate.

[0047] The first-order optimization algorithm will record historical gradient information, that is, in addition to the memory space required by the automatic differentiation algorithm, additional memory overhead is required. By exchanging memory for time, the model convergence is accelerated, which makes it impossible to use only one times the inference memory overhead for model training under specific conditions. For example, the model update method of momentumSGD is w←w-η(g+αb), where w is the model weight, η is the learning rate, g is the gradient estimate, α is the momentum parameter, the momentum parameter is greater than 0 and less than 1, and b is the momentum, which is a vector with the same length as g. Momentum is actually a linear combination of historical gradient information. Vector b is calculated by g many times before, so additional memory space needs to be opened up to store the gradient estimates g many times before. The embodiment of the present application limits the queue length of the seed queue and the projection queue, ignores the historical gradient information that is too old, and uses the seed-projection pair of the seed queue and the projection queue to represent the historical gradient information of a limited number of steps, so as to reduce the storage space required for the historical gradient information to an extremely low level.

[0048] See also Figure 5 In some embodiments, step S160 may include but is not limited to steps S510 to S530: Step S510, determining momentum decision parameters; Step S520: if the momentum decision parameter indicates to use the momentum variable to update the model parameters, then the model parameters of the preset large language model are preliminarily updated according to the preset learning rate, random seed and projection parameters; Step S530, the model parameters after the preliminary update are updated again according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate, until the current iteration round reaches the preset iteration number threshold, and the target large language model is obtained.

[0049] In step S510 of some embodiments, a momentum decision parameter is obtained, and the momentum decision parameter is used to indicate whether the model training of the current iteration round uses Nesterov momentum. The momentum decision parameter is a Boolean variable, and its value is 0 or 1. If the momentum decision parameter is 1, it means that the model training of the current iteration round uses Nesterov momentum, otherwise Nesterov momentum is not used.

[0050] In step S520 of some embodiments, if the momentum decision parameter indicates that the model parameters are updated using the momentum variable, the momentum decision parameter is 1, and an initial Gaussian pseudo-random vector is generated according to the random seed, and the initial Gaussian pseudo-random vector is multiplied by the projection parameter to obtain an initial zero-order gradient estimate. The model parameters of the preset large language model are preliminarily updated according to the preset learning rate and the initial zero-order gradient estimate. The preset learning rate is used to control the step size of the model parameter update during the model training process. The formula for the preliminary update is defined as: w←w-ηg, where w is the model parameter of the preset large language model, η is the preset learning rate, and g is the initial zero-order gradient estimate. If the momentum decision parameter indicates that the model parameters are not updated using the momentum variable, step S520 is skipped, and referring to step S530, the model parameters of the preset large language model are updated according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold, and the target large language model is obtained.

[0051] In step S530 of some embodiments, the preset momentum parameter is a hyperparameter for accelerating the gradient descent process, and the preset momentum parameter is greater than 0 and less than 1. The seed queue is represented as slist, and it is assumed that the seed queue contains h random seeds. According to the first-in-first-out principle, the jth random seed is obtained from the seed queue as the target random seed slist[j], and the model parameters initially updated in the current iteration round are updated again according to the preset learning rate, the preset momentum parameter, the position identifier j of the target random seed slist[j] in the seed queue, and the target zero-order gradient estimate corresponding to the target random seed slist[j], to obtain the intermediate model parameters updated based on the current target random seed. Continue to obtain the next target random seed slist[j+1] from the seed queue, and estimate the target zero-order gradient corresponding to the target random seed slist[j+1] according to the preset learning rate, preset momentum parameter, the position j+1 of the target random seed slist[j+1] in the seed queue, and the target random seed slist[j+1], and update the intermediate model parameters based on the previous target random seed. Update and output intermediate model parameters Until the intermediate model parameters based on the last target random seed output in the seed queue are obtained , the model parameters of the current iteration round are updated, and the iterative update process of step S120 to step S160 is repeated until the current iteration round reaches the preset iteration number threshold, and the target large language model is obtained. It should be noted that each iteration round is to update the model parameters of the preset large language model updated in the previous iteration round.

[0052] In the above steps S510 to S530, the model training process is controlled by the momentum decision parameter to accelerate the model convergence and improve the model training efficiency. At the same time, the model parameters are iteratively updated by each random seed stored in the seed queue, which can ignore the historical gradient information that is too old, and there is no need to save the historical gradient information. Compared with the first-order algorithm, the memory space required for model training is saved.

[0053] See also Figure 6 In some embodiments, step S530 may include but is not limited to steps S610 to S650: Step S610, determining the exponent of the preset momentum parameter according to the position identifier of the target random seed in the seed queue, performing exponential calculation on the position identifier and the preset momentum parameter to obtain an initial update parameter; Step S620, multiplying the preset learning rate, the initial update variable and the target zero-order gradient estimate to obtain a reference update parameter; Step S630, updating the initially updated model parameters again according to the reference update parameters.

[0054] In step S610 of some embodiments, the dequeue identifier is determined according to the queue length of the seed queue and the position identifier of the target random seed in the seed queue. If the position identifier is j and the queue length is h, the dequeue identifier is h-j+1. The seed queue follows the first-in-first-out principle. To facilitate the entry and exit of random seeds, the position identifier of the recently added random seed in the seed queue is smaller and the dequeue identifier is larger. The position identifier of the random seed added to the seed queue in the current iteration round is 1, and the dequeue identifier is h.

[0055] The dequeue flag is used as the exponent of the preset momentum parameter, and the exponential calculation of the dequeue flag and the preset momentum parameter is performed to obtain the initial update parameter. The initial update parameter is expressed as: , in, is a preset momentum parameter, which is greater than 0 and less than 1; h is the queue length of the seed queue; j is the position identifier of the target random seed in the seed queue.

[0056] In step S620 of some embodiments, a preset learning rate, an initial update variable, and a target zero-order gradient estimate are multiplied to obtain a reference update parameter.

[0057] In step S630 of some embodiments, the model parameters after the initial update are subtracted from the reference update parameters to re-update the model parameters after the initial update. The re-updated formula is defined as: , Among them, w is the model parameter after preliminary update; is the preset learning rate; is the preset momentum parameter; h is the queue length; p is the target projection parameter; z is the target Gaussian pseudo-random vector.

[0058] Momentum b is a linear combination of historical gradient information. The momentum of the embodiment of the present application can be expressed as b=g+α*g_1+α^2*g_2+…+α^k*g_k+…+α^h*g_h, where g_k represents the gradient information recorded when the gradient is k steps away from the current step, ^ represents exponential operation, and * represents multiplication operation. Note that 0<α<1, and the influence of gradient information that is too old on momentum b will actually gradually decrease. Therefore, when the historical window length h is not too small, the historical gradient reconstructed by the seed queue and projection queue will not have a large error with the method of explicitly recording momentum. At the same time, when using the zero-order optimization algorithm to update the model parameters, it is no longer necessary to open up additional memory space to record momentum, but only h matching seeds and projections need to be recorded, which greatly reduces the memory overhead of model training.

[0059] Through the above steps S610 to S630, the influence of historical gradient information that is too old on model training is reduced, and the historical gradient can be determined by recording the random seed and projection parameters without opening up additional storage space to store the historical gradient. While ensuring the model classification performance, the memory space required for model training is reduced.

[0060] Figure 7 is a flowchart of a text classification method provided by an embodiment of the present application, which may include but is not limited to steps S710 to S720: Step S710, obtaining the target text; Step S720, classifying the target text using the target large language model to obtain a target text category.

[0061] In step S710 of some embodiments, a target text is obtained, where the target text is the text to be classified.

[0062] In step S720 of some embodiments, the target large language model trained by the above-mentioned model training method performs text classification on the target text and outputs the target text category.

[0063] In the above steps S710 to S720, the text category of the target text is predicted by a large language model, which can accurately understand the text semantics of the target text and improve the accuracy of text classification.

[0064] See also Figure 8 The present application also provides a text classification device that can implement the above text classification method. The text classification device includes: An acquisition module 810 is used to acquire a target text; The classification module 820 is used to classify the target text using the target large language model to obtain the target text category.

[0065] The specific implementation of the text classification device is basically the same as the specific implementation of the above-mentioned text classification method, and will not be repeated here.

[0066] The present application also provides a model training device, which can implement the above-mentioned model training method, and the model training device includes: Initialization module, used to initialize the seed queue and projection queue; A first acquisition module is used to acquire a random seed, a natural language text, and a text category label of the natural language text in a current iteration round; A determination module, used to determine projection parameters of a preset large language model according to a random seed, a natural language text, and a text category label; Added modules to add random seeds to the seed queue and projection parameters to the projection queue; The second acquisition module is used to sequentially acquire a target random seed from the seed queue, generate a target Gaussian pseudo-random vector according to the target random seed, acquire projection parameters in the projection queue according to the target random seed, obtain target projection parameters, and multiply the target Gaussian pseudo-random vector and the target projection parameters to determine the target zero-order gradient estimate; The update module is used to update the model parameters of the preset large language model according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate for each iteration round, until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model; wherein the target large language model is used to perform text classification on the input target text.

[0067] In some embodiments, the determination module is further configured to: Generate an initial Gaussian pseudo-random vector according to a random seed; Performing forward perturbation on model parameters of a preset large language model according to an initial Gaussian pseudo-random vector and a preset perturbation step size to obtain forward perturbation parameters; Determine a first loss based on the forward perturbation parameters, the natural language text, and the text category label; Performing backward perturbation on the forward perturbation parameter according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain the backward perturbation parameter; The projection parameters of the preset large language model are determined according to the first loss, the backward perturbation parameter, the natural language text and the text category label.

[0068] In some embodiments, the determination module is further configured to: Determine a second loss based on the backward perturbation parameter, the natural language text, and the text category label; A projection parameter of a preset large language model is determined according to the first loss, the second loss, and a preset perturbation step size.

[0069] In some embodiments, the adding module is further used to: Get the queue length of the seed queue; both the seed queue and the projection queue follow the first-in-first-out principle; If the queue length is less than the preset length threshold, the random seed is added to the seed queue and the projection parameter is added to the projection queue; If the queue length is equal to the preset length threshold, the earliest random seed is popped out from the seed queue and the earliest projection parameter is popped out from the projection queue according to the first-in-first-out principle, and the random seed is added to the seed queue and the projection parameter is added to the projection queue.

[0070] In some embodiments, the update module is further used to: Determine momentum decision parameters; If the momentum decision parameter indicates that the momentum variable is used to update the model parameters, the model parameters of the preset large language model are preliminarily updated according to the preset learning rate, random seed and projection parameters; The model parameters after the initial update are updated again according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold to obtain the target large language model.

[0071] In some embodiments, the update module is further used to: Determine the exponent of the preset momentum parameter according to the position identifier of the target random seed in the seed queue, perform exponential calculation on the position identifier and the preset momentum parameter, and obtain the initial update parameter; Multiply the preset learning rate, the initial update variable, and the target zero-order gradient estimate to obtain the reference update parameter; The initially updated model parameters are updated again according to the reference update parameters.

[0072] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned model training method or text classification method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0073] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the model training method or text classification method of the embodiments of this application; Input / output interface 930, used to implement information input and output; Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 ); The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0074] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned model training method or text classification method.

[0075] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0076] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0077] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0078] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0079] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0080] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0081] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0082] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0083] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0084] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0086] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model training method, characterized in that: The method comprises: Initialize the seed queue and projection queue; In the current iteration round, a random seed, a natural language text, and a text category label of the natural language text are obtained; Determining projection parameters of a preset large language model according to the random seed, the natural language text, and the text category label; Adding the random seed to the seed queue, and adding the projection parameter to the projection queue; Obtain a target random seed from the seed queue in sequence, generate a target Gaussian pseudo-random vector according to the target random seed, obtain projection parameters in the projection queue according to the target random seed, obtain target projection parameters, and multiply the target Gaussian pseudo-random vector and the target projection parameters to determine a target zero-order gradient estimate; For each iteration round, the model parameters of the preset large language model are updated according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model; wherein the target large language model is used to perform text classification on the input target text.

2. The method according to claim 1, characterized in that The step of determining the projection parameters of the preset large language model according to the random seed, the natural language text, and the text category label includes: Generate an initial Gaussian pseudo-random vector according to the random seed; Performing forward perturbation on model parameters of the preset large language model according to the initial Gaussian pseudo-random vector and a preset perturbation step size to obtain forward perturbation parameters; Determining a first loss according to the forward perturbation parameter, the natural language text, and the text category label; Performing backward perturbation on the forward perturbation parameter according to the initial Gaussian pseudo-random vector and the preset perturbation step size to obtain a backward perturbation parameter; Determine projection parameters of a preset large language model according to the first loss, the backward perturbation parameter, the natural language text, and the text category label.

3. The method according to claim 2, characterized in that The step of determining the projection parameters of the preset large language model according to the first loss, the backward perturbation parameter, the natural language text, and the text category label includes: Determining a second loss according to the backward perturbation parameter, the natural language text, and the text category label; A projection parameter of a preset large language model is determined according to the first loss, the second loss, and the preset perturbation step size.

4. The method according to claim 1, characterized in that: The adding the random seed to the seed queue and the adding the projection parameter to the projection queue comprises: Obtaining the queue length of the seed queue; both the seed queue and the projection queue follow the first-in-first-out principle; If the queue length is less than a preset length threshold, adding the random seed to the seed queue and adding the projection parameter to the projection queue; If the queue length is equal to the preset length threshold, the earliest random seed is popped out from the seed queue and the earliest projection parameter is popped out from the projection queue according to the first-in-first-out principle, and the random seed is added to the seed queue and the projection parameter is added to the projection queue.

5. The method according to any one of claims 1 to 4, characterized in that: The method updates the model parameters of the preset large language model according to the preset learning rate, the preset momentum parameter, the position identifier of each of the target random seeds in the seed queue, and the corresponding target zero-order gradient estimation until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model, including: Determine momentum decision parameters; If the momentum decision parameter indicates that the model parameters are updated using a momentum variable, the model parameters of the preset large language model are preliminarily updated according to the preset learning rate, the random seed and the projection parameter; The model parameters after the preliminary update are updated again according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue and the corresponding target zero-order gradient estimate until the current iteration round reaches the preset iteration number threshold, thereby obtaining the target large language model.

6. The method according to claim 5, characterized in that The step of re-updating the initially updated model parameters according to the preset learning rate, the preset momentum parameter, the position identifier of each target random seed in the seed queue, and the corresponding target zero-order gradient estimate comprises: Determine the exponent of the preset momentum parameter according to the position identifier of the target random seed in the seed queue, perform exponential calculation on the position identifier and the preset momentum parameter, and obtain an initial update parameter; Multiplying the preset learning rate, the initial update variable and the target zero-order gradient estimate to obtain a reference update parameter; The model parameters after the initial update are updated again according to the reference update parameters.

7. A text classification method, characterized in that: The method comprises: Get the target text; The target text is classified by a target large language model to obtain a target text category; wherein the target large language model is trained by the model training method according to any one of claims 1 to 6.

8. A text classification device, characterized in that: The device comprises: An acquisition module is used to acquire the target text; A classification module is used to perform text classification on the target text through a target large language model to obtain a target text category; wherein the target large language model is trained according to the model training method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method according to any one of claims 1 to 6 or the text classification method according to claim 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model training method according to any one of claims 1 to 6 or the text classification method according to claim 7 is implemented.

Images