Language model dynamic structuring efficient fine-tuning method and electronic device

Abstract

The application discloses a language model dynamic structuring efficient fine-tuning method and electronic equipment, wherein the method comprises the following steps: obtaining a natural language model needing fine-tuning; obtaining a data set of a natural language downstream task, and designing a prompt sentence for guiding the model to understand the downstream task; and fine-tuning the natural language model by using the data set and the prompt sentence; wherein the fine-tuning process comprises the following steps: updating the parameters of the model by using a zero-order optimization method; adjusting a forward propagation process by using a structured pruning scheme; dynamically adjusting a pruning structure by using a genetic algorithm and a random selection method; and determining exploration or the best pruning scheme by using a reinforcement learning strategy. The application introduces a structured pruning, a reinforcement learning and a genetic algorithm to solve the problems of large calculation overhead and long training time of the zero-order optimization process. The application can reduce the calculation overhead of the zero-order optimization on all models and tasks while maintaining the performance of the model, and can be widely applied to the field of natural language model fine-tuning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text processing, pruning and zero-order optimization, and particularly relates to a language model dynamic structured efficient fine-tuning method and an electronic device. BACKGROUND

[0002] Language models are excellent in understanding and generating complex tasks and are applied more and more widely. In order to make pre-trained language models quickly applied to downstream tasks, fine-tuning technology has become a highly efficient and universal solution. However, with the increase of the parameter size and complexity of the language model, the computing cost related to fine-tuning rises rapidly. The large amount of resource demand poses a very big obstacle to the application and research of the language model. Therefore, it is of great significance to design a memory and computing efficient fine-tuning method for the low-cost application of the language model.

[0003] In order to reduce the computing resource consumption when fine-tuning a large model, the commonly used methods include model level solutions and optimization level solutions. At the model level, structured pruning or selective updating of part of the parameters is a commonly used effective method. These solutions can reduce the memory consumption by reducing the storage state of the first-order optimizer, but the first-order optimizer still needs to store a large amount of memory for gradient, momentum and other information. At the optimizer level, a zero-order optimizer is proposed to replace the first-order optimizer. The zero-order optimizer uses an approximate gradient and does not need to store the gradient information of each parameter, thus greatly saving the memory consumption in the model fine-tuning process. However, since the zero-order gradient is obtained by differential algorithm approximation, the convergence speed is very slow. How to design an optimization scheme to reduce the memory consumption and computing consumption in the fine-tuning process is a research direction worthy of in-depth exploration. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a language model dynamic structured efficient fine-tuning method based on zero-order optimization, an electronic device and a medium.

[0005] The first technical solution adopted by the present application is:

[0006] A language model dynamic structured efficient fine-tuning method, comprising the following steps:

[0007] Obtaining a natural language model that needs to be fine-tuned; obtaining a data set of a natural language downstream task, and designing a prompt sentence (prompt) for guiding the model to understand the downstream task;

[0008] Fine-tuning the natural language model by using the data set and the prompt sentence;

[0009] In the fine-tuning process, it includes:

[0010] updating parameters of the model using a zeroth order optimization method;

[0011] adjusting the forward propagation process using a structured pruning scheme;

[0012] dynamically adjusting the pruning structure using a genetic algorithm and a random selection method;

[0013] using a reinforcement learning strategy to decide to explore or execute the best pruning scheme;

[0014] In the test, the fine-tuned language model is used to obtain the test results on the downstream task.

[0015] Further, the natural language model is a large artificial intelligence neural network based on a Transformer structure, including a model based on an encoder structure (such as the BERT series model) and a model based on a decoder structure (such as the Llama, OPT series model, etc.). The natural language model is a language model with certain language understanding capabilities.

[0016] Further, the model based on the encoder structure is composed of L cascaded encoder layers, each of which is composed of cascaded self-attention modules, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network, and a layer normalization processing module.

[0017] The model based on the decoder structure is composed of L cascaded decoder layers, each of which is composed of cascaded self-attention modules, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network, and a layer normalization processing module.

[0018] Further, the data set of the natural language downstream task includes classification tasks, multiple-choice tasks, and generation tasks, and the data is composed of existing language texts.

[0019] Further, the prompt sentence is a sentence composed of a text of the same language category as the downstream task data set, which is targeted and suggestive.

[0020] Further, the zeroth order optimization method refers to a zeroth order optimization method capable of fine-tuning a language model containing a large number of parameters.

[0021] After obtaining the parameters θ that need to be updated in the fine-tuning step, the zeroth order optimizer estimates the gradient of the model, and updates the model parameters using this gradient and the set perturbation information.

[0022] Wherein, the zeroth order optimization is a parameter optimization method without explicit calculation of the gradient information of the objective function

[0023] Further, the SPSA method is used to estimate the zero-order gradient, and a zero-order stochastic gradient descent (ZO-SGD) scheme is used to update the model parameters;

[0024] In the SPSA method, a normal distribution matrix z consistent with the size of the model parameters is obtained, and the target function L in the fine-tuning process of the model is used to estimate the first-order gradient by slightly perturbing the model parameters in a random direction and then calculating the difference between the predicted values of the target function after two forward propagations under the condition of a small batch of data B:

[0025]

[0026] In the formula, μ is a perturbation scaling factor, T represents transposition, denotes the gradient symbol;

[0027] The parameters of the model are updated using the zero-order stochastic gradient descent scheme as follows:

[0028] The expression for updating the model parameters using the zero-order gradient at the t-th time is:

[0029]

[0030] In the formula, θ t is the model parameter set at time t, B t is the training data set at time t, and η is the learning rate.

[0031] Further, the structured pruning scheme is used to adjust the forward propagation process, including:

[0032] The structured sparse scheme is used to divide the model parameters, and matrix operations can be used to add perturbations to the model. For example, the layer structure of a language model is used as the most basic pruning structure, and the parameters of the layer that does not perform the pruning scheme are calculated by forward propagation, and at the same time, a perturbation term can be added to the parameter matrix of the layer by performing matrix addition operation. Using modules or matrices as the basic structure of pruning is also an effective scheme.

[0033] Further, the genetic algorithm and the random selection method are used to dynamically adjust the pruning structure, including:

[0034] The structure of the model in each training step is adjusted using a genetic algorithm. First, in the first m steps, the first generation of the species is obtained by a random basic structure discard strategy, such as using the layers of the model as the basic structure, and the sequence number of the discarded layer each time constitutes the gene of each individual. After m+1 steps, the genetic strategy is used to update the species. For example, using a simple genetic strategy, all individuals in the species are used as parents to generate a new offspring, the genes of the offspring are generated based on the number distribution probability of all parent genes, and the occurrence number of each gene is set to be at least 1, and the mutation probability of the population is maintained, so as to maintain an excellent population. The method is not limited to using the above simple genetic algorithm.

[0035] Further, the use of reinforcement learning strategy to determine exploration or optimal pruning scheme, including:

[0036] The exploration and utilization of the layer sparsity scheme are balanced using the reinforcement learning scheme, and the efficient training process benefit is maximized. For example, an e-greedy algorithm is used, with an e probability to perform search at the t training step, and a predetermined strategy is used to generate a set of unexplored layer sparsity schemes, and with a 1-e probability, the optimal scheme in the already explored sparsity scheme is used. Since the training is a dynamic process, the benefit of each sparsity scheme is dynamically updated in the greedy strategy.

[0037] Further, the benefit of each sparsity scheme is dynamically updated, including:

[0038] The negative value of the loss on the small batch data of the training is used as the benefit, and the benefit of using the specified structured pruning scheme is gradually improved as the training proceeds. In order to synchronize the benefits of all structured pruning schemes as the training process proceeds, a discount factor γ is introduced, and in the next step, the benefits of all schemes stored in the buffer are multiplied by γ.

[0039] The second technical solution adopted by the present application is:

[0040] An electronic device, the electronic device includes a processor and a memory, the memory has at least one instruction, at least one program, a code set or an instruction set stored therein, the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to realize a language model dynamic structured efficient fine-tuning method as described above.

[0041] The third technical solution adopted by the present application is:

[0042] A computer readable storage medium, the storage medium stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to realize a language model dynamic structured efficient fine-tuning method as described above.

[0043] The fourth technical solution adopted by the present application is:

[0044] A computer program product or computer program, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer readable storage medium. The processor of the computer equipment can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer equipment executes the above method.

[0045] The beneficial effects of the present application are: the present application proposes an innovative dynamic efficient language model fine-tuning scheme, which can be applied to the fine-tuning of any pre-trained language model on any downstream task. In order to realize memory efficiency, the present application uses zero-order optimization method to update model parameters, which greatly saves the storage consumption of parameter state. The scheme of structured pruning reduces the calculation overhead in the fine-tuning process. At the same time, in order to reduce the hard error brought by structured pruning, the present application proposes a dynamic pruning structure search strategy, which uses epsilon-greedy algorithm and genetic algorithm to update the pruning structure. Under the combination of the above scheme, the present application realizes the zero-order dynamic structured fine-tuning scheme with memory and calculation efficiency, which can effectively reduce the resource consumption in the language model fine-tuning process, while ensuring the performance of fine-tuning. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0047] Figure 1 is a schematic diagram of a structured sparse scheme in an embodiment of the present application;

[0048] Figure 2 is a schematic diagram of dynamically updating the benefits of each sparse scheme in an embodiment of the present application;

[0049] Figure 3 is a greedy algorithm flowchart / genetic algorithm flowchart in an embodiment of the present application. DETAILED DESCRIPTION

[0050] Embodiments of the present application are described below in detail with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below are exemplary only, and are used to explain the present application, and are not to be understood as limiting the present application. For the step numbers in the following embodiments, they are only set for the convenience of the description of the explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0051] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by the upper, lower, front, rear, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0052] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0053] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0054] Term explanation:

[0055] Language Model: Language model is a deep learning model that can process natural language. It learns the statistical rules and semantic representations of language by pre-training on large-scale corpus. Large language models generally have high language understanding and generation capabilities and can be used for various natural language processing tasks.

[0056] Zero-Order Optimizer: Zero-order optimization refers to optimization methods that do not require gradient information. Generally, it refers to methods that estimate the gradient of the objective function with respect to parameter updates based on parameter adoption and difference. Zero-order optimizer refers to an optimizer that uses zero-order optimization methods to update model parameters.

[0057] First-Order Optimizer: First-order optimizer refers to an optimizer that calculates gradient information and Hessian information of model parameters using strategies such as back propagation, and updates model parameters using this information.

[0058] Embodiment 1

[0059] As Figure 1 shown, the embodiment provides a language model dynamic structured efficient fine-tuning method based on zero-order optimization, which includes a model sparsification scheme for structure search, a reinforcement learning scheme for strengthening the benefits of fine-tuning process, a genetic algorithm for sparse structure construction, a sparse zero-order optimization scheme, a sparse zero-order stochastic gradient descent scheme, and a zero-order structured fine-tuning training strategy. The model sparsification scheme for structure search can reduce floating point calculations in the model optimization process and improve computational efficiency; reinforcement learning can effectively balance the exploration of model sparse structure exploration and the consumption of fine-tuning process, and reduce the performance decline of the model relative to the ordinary zero-order optimization process; the genetic algorithm for sparse structure construction can reduce unnecessary exploration in the structure search of reinforcement learning, narrow the exploration range, and improve the efficiency of fine-tuning; the sparse zero-order optimization scheme can avoid the process of calculating gradient back propagation, and relative to the first-order optimizer, it can reduce the storage of optimizer state, achieving the purpose of saving memory overhead; the sparse zero-order stochastic gradient descent scheme can effectively adjust all parameters while reducing the number of updated parameters, reducing the time consumption of fine-tuning; through the combination of the above schemes, a zero-order structured fine-tuning training strategy is obtained, which can greatly reduce the memory and time consumption in the fine-tuning process of the language model, at the same time, an effective structure pruning scheme for downstream tasks can be obtained, and the importance distribution of model parameters can be obtained. The method specifically includes the following steps:

[0060] S1, obtaining the data set of the specified natural language downstream task, designing a suitable prompt to construct effective data for zero-order fine-tuning; obtaining the natural language model that needs to be fine-tuned.

[0061] In this embodiment, the natural language downstream task dataset is selected as SST-2, and the natural language model to be fine-tuned is selected as OPT-13B. For the SST-2 dataset, a suitable prompt is designed, and here the form of the prompt is set as {text}It was {terrible or great}, {text} place needs to fill in the text information in the SST-2 dataset, and {terrible or great} place needs to fill in the label information in the SST-2 dataset, using one of terrible or great. For the OPT-13B model, it is a pre-trained decoder-based natural language model and has certain ability for natural language understanding and expression. In Table 1, more prompt designs for other natural language datasets that can be used for zero-order optimization are provided.

[0062] Table 1 Prompt for other natural language downstream task datasets in the present application

[0063]

[0064] S2, initialize hyperparameters.

[0065] There are a large number of hyperparameters in this method, and in this embodiment, the hyperparameter settings adopted are: the number of layer sparsities n = 6; the set a of layer information is obtained in a random initialization manner; the size m of the empty buffer sequence is 100; the initial exploration probability ∈ = 0.9; the minimum exploration probability β = 0.3; the perturbation scaling factor μ = 1e-3; the discount factor γ = 0.9; the decay factor α = 0.99; the total training step T = 20000; the learning rate l = 1e-7; the size of the training dataset is set to 1000; and the batch data size B = 16.

[0066] S3, the ∈-greedy strategy decides whether to explore or exploit.

[0067] The scheme of the greedy strategy is to first generate a random number ρ from 0 to 1, and judge the size of the random number ρ and ∈, when ρ < ∈, the exploration process is executed, and when ρ > ∈, the exploitation process is executed. With the deepening of the exploration of the layer sparsity structure, there will be layer importance information in the buffer, so the exploration probability is gradually reduced in this embodiment. After each execution of the greedy strategy, the exploration probability is updated as ∈ = max(β, α∈), and ∈ will gradually decrease to the minimum exploration probability β, which is achieved around 2000 cycle steps. This setting can quickly explore the entire layer sparsity structure space. Figure 1 The layer sparsity scheme and the reason for acceleration are shown in detail in the foregoing.

[0068] S4, genetic algorithm updates the structured sparse scheme. The buffer obtains the highest return scheme.

[0069] In Figure 2 The selection strategy of the structured sparse scheme is shown in the detailed description. When the ∈-greedy algorithm selects the exploration stage, this embodiment updates the sparse layer structure information a using a simplified genetic algorithm. The detailed process is as follows: when the information stored in the buffer is less than m, the buffer is not full, and a layer sparse set a containing n elements is generated in a random sampling manner; when the buffer is full, the occurrence probability of all layers in the buffer is counted, and n layer serial numbers are extracted from 1,…,N without repetition according to the probability to form a. When the ∈-greedy algorithm selects the utilization stage, the sparse scheme a with the highest return in the buffer is selected. Figure 3 The layer sparsification strategy update scheme in each step is shown in detail in the detailed description.

[0070] S5, according to the structured coefficient scheme, +μ perturbation is added to the model, and l + .

[0071] In this embodiment, the target loss of the model in the training process is where K is the total length of the sequence, P(x k | x1, x2,…, x k-1 ) is the probability of the kth word predicted by the model under the given previous words. By calculating the negative log-likelihood of each word, the loss is minimized to increase the ability of the model. In the case of obtaining the layer sparse set a, first, according to a, the parameter set of the layer to be skipped is obtained:

[0072]

[0073] A random seed s is determined to generate a random Gaussian perturbation consistent with the size of the adjustment parameter, and a scaling factor of +1 is used for scaling:

[0074]

[0075]

[0076] S6, according to the structured coefficient scheme, -2μ perturbation is added to the model, and l - .

[0077] The random seed s is reset to generate a random Gaussian perturbation consistent with the size of the adjustment parameter, and a scaling factor of -2 is used for scaling:

[0078]

[0079]

[0080] S7, add +u perturbation to the model according to the structured coefficient scheme, restore the model parameters.

[0081] Reset the random seed s, generate a random Gaussian perturbation consistent with the size of the adjustment parameter, and scale it using a scaling factor of +1:

[0082]

[0083] S8, estimate the zero-order gradient project_grad = (l + +l - ) / 2.

[0084] In this implementation, the SPSA method is used to estimate the first-order gradient. The difference is estimated according to the loss values obtained after two perturbations, and the first-order estimated gradient is obtained:

[0085]

[0086] S9, update the model parameters using the zero-order random gradient descent (ZO-SGD) scheme.

[0087] Reset the random seed s, and update the sparse layer parameters using the random gradient descent method according to the zero-order gradient project_grad obtained in step S8:

[0088]

[0089] S10, maintain the update sparse scheme and the reward buffer.

[0090] The sequence update function UpdateSeq(a, r, project_grad) is used to update the buffer, where a is the set of pruned layer information, r is the reward score, and project_grad is the zero-order estimated gradient:

[0091] For all :

[0092]

[0093] Insert [a, r] into the position that satisfies .

[0094] Loop steps S3-S10 for 20000 times to obtain the model with the minimum loss on the test set in this process.

[0095] S11, use the trained language model to obtain the results on the downstream task test data set.

[0096] To verify the effectiveness of the present application, after the training is completed on the SST-2 training set, the test set of SST-2 is tested, and the accuracy is used as the evaluation standard, that is, the proportion of correct classification on all test samples. In addition to the SST-2 data set, in order to further prove the effectiveness, the present embodiment also tests other different size models and data sets. And compared with other methods. The present application has similar performance to ordinary zero-order optimization method in improving the calculation speed by 10%+. The performance of the present application is shown in Table 2, in which DROP uses F1 score.

[0097] Table 2 Performance of the present application on Llama-7B, OPT-13B models using different downstream task data sets

[0098]

[0099]

[0100] In summary, the present embodiment has the following advantages and beneficial effects compared with the prior art:

[0101] (1) The method of the present embodiment includes a dynamic structured sparse scheme, which effectively reduces the computational expenditure of zero-order optimization and reduces the training time.

[0102] (2) The method of the present embodiment also includes a sparse layer structure search scheme, which can maintain the training effect while searching for relatively unimportant layers for model inference during the training process.

[0103] (3) By adjusting the training process through a reinforcement learning algorithm, the benefits of the training process and the benefits of layer structure exploration can be balanced, and the training task can be completed while maintaining the training effect.

[0104] (4) The dynamic structured efficient fine-tuning method based on zero-order optimization proposed in the present embodiment can be combined with other efficient fine-tuning (PEFT) schemes to further improve the training speed of zero-order optimization, and has wide application value.

[0105] Embodiment 2

[0106] A dynamic structured efficient fine-tuning method based on zero-order optimization includes the following steps:

[0107] Step 1: Obtain a natural language downstream task data set , based on the task type, construct a prompt, and determine a target loss function .

[0108] Step 2: Obtain a pre-trained language model, obtain the number of layers information N of the model, and all parameters of the model are θ.

[0109] Step 3: According to the data set and prompt, fine-tune the language model, update the parameters θ. The fine-tuning process is as follows:

[0110] Set a learning rate scheduling set {η t} related to the training step t; initialize a layer information set a, which contains any n layer numbers in 1,…,N; initialize an empty buffer sequence with size m, where the elements are a data pair, and the data pair consists of a layer information set and a score; initialize an exploration probability 0≤∈<1. Define a minimum exploration probability 0≤β<1; define a perturbation scaling factor μ; define a discount factor γ; define a decay factor α.

[0111] 1) Define a perturbation function Perturb(θ,μ,s), where θ is the parameter set, μ is the scaling factor, and s is a random seed:

[0112] Set the global random seed to s;

[0113] For any θ i ∈θ:

[0114] Generate a random normal distribution perturbation

[0115] Update the parameters θ i =θ i +μz;

[0116] Return the updated parameters θ.

[0117] 2) Define a sequence update function UpdateSeq(a,r,project_grad), where a is the pruned layer information set, r is the reward score, and project_grad is the zero-order estimated gradient:

[0118]

[0119] 3) Process all samples in the data set , concatenate the prompt;

[0120] For the training process from step t from 1 to T:

[0121] According to a, obtain the parameter set of the layers that need to be skipped

[0122] Randomly select b samples from the data set to form a small training batch ;

[0123] Randomly initialize a random seed s;

[0124] Apply a random perturbation of size +μ to the model parameter subset θ' and update the parameter subset θ' = Perturb(θ, μ, s);

[0125] Calculate the loss of the model at this time

[0126] Apply a random perturbation of size -2μ to the model parameter subset θ' and update the parameter subset θ' = Perturb(θ, -2μ, s);

[0127] Calculate the loss of the model at this time

[0128] Apply a random perturbation of size -μ to the model parameter subset θ' and update the parameter subset θ' = Perturb(θ, -2μ, s);

[0129] Calculate the gradient using difference estimation

[0130] Reset the global random seed to s;

[0131] Update the model parameters, for any θ i ∈θ:

[0132] Generate a random normal distribution perturbation

[0133] Update the parameters θ i = θ i -η×project_grad×z;

[0134] Update the set of layer information

[0135] Obtain the fine-tuned language model parameters θ.

[0136] To sum up, the embodiment provides a language model dynamic structuring efficient fine-tuning method based on zero-order optimization, acquires a dataset of a specified natural language downstream task, designs a suitable prompt to construct data effective for zero-order fine-tuning; acquires a natural language model that needs to be fine-tuned; initializes hyperparameters; an ∈-greedy strategy decides whether to explore or utilize; a genetic algorithm updates a model structured sparse scheme or a buffer to obtain a scheme with the highest return; according to the structured coefficient scheme, +μ perturbation is added to the model, and l + is calculated; according to the structured coefficient scheme, -2μ perturbation is added to the model, and l -; add + mu perturbation to the model according to the structured coefficient scheme, restore the model parameters; estimate the zero-order gradient project_grad = (l + +l - ) / 2 mu; update the model parameters using ZO-SGD; maintain the updated sparse scheme and the reward buffer; loop the steps; obtain the results on the downstream task test dataset by using the trained language model. The structured pruning, reinforcement learning and genetic algorithm are introduced to solve the problems of large computational overhead and long training time in the zero-order optimization process. The method of the embodiment can reduce the computational overhead of zero-order optimization on all models and tasks while maintaining the model performance, and can be widely applied in the field of natural language model fine-tuning.

[0137] Embodiment 3

[0138] The embodiment of the application also provides an electronic device, which comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method as shown in Figure 1 .

[0139] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing each of the above-mentioned method embodiments, etc.; and the data storage area can store data created according to the use of the server, etc.

[0140] The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Alternatively, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be realized by a single chip.

[0141] Since the electronic device is an electronic device corresponding to the language model dynamic structuring efficient fine-tuning method of the embodiments of the application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be repeated.

[0142] Embodiment 4

[0143] The embodiments of the application also provide a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the method as shown in Figure 1

[0144] ​Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0145] Since the storage medium is a storage medium corresponding to the language model dynamic structured efficient fine-tuning method of the embodiments of the present application, and the problem solving principle of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0146] Embodiment 5

[0147] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the language model dynamic structured efficient fine-tuning method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein, the executable computer program codes or "codes" for executing various embodiments can be written in high-level programming languages such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0148] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, can be used to implement the hardware: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth, or a combination of them.

[0149] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0150] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made in accordance with the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for dynamically structuring efficient fine-tuning of a language model, characterized in that, The method comprises the following steps: acquiring a natural language model that needs to be fine-tuned; acquiring a dataset of a natural language downstream task, and designing a prompt sentence for guiding the model to understand the downstream task; fine-tuning the natural language model using the dataset and the prompt sentence; In the fine-tuning process, the following steps are included: updating the parameters of the model using a zero-order optimization method; adjusting the forward propagation process using a structured pruning scheme; dynamically adjusting the pruning structure using a genetic algorithm and a random selection method; deciding to explore or execute the best pruning scheme using a reinforcement learning strategy; The fine-tuning process specifically includes: S3、 - The greedy strategy decides whether to explore or exploit: the solution implemented by the greedy strategy is to first generate a random number between 0 and 1 , judge the size of the random number and the exploration probability , when , the exploration process is executed, and when , the exploitation process is executed; after each execution of the greedy strategy, the exploration probability is updated, which gradually decreases to the minimum exploration probability in the exploration process; S4, genetic algorithm updates the structured sparse scheme of the model, and the buffer obtains the highest return scheme: when - the greedy algorithm selects the exploration stage, and the genetic algorithm updates the sparse layer structure information : when the information stored in the buffer is less than , the buffer area is not full, and a layer sparse set containing elements is generated by random sampling ; when the buffer is full, the occurrence probability of all layers in the buffer is counted, and layer serial numbers are extracted from all layer serial numbers in the buffer without repetition to form ; when - the greedy algorithm selects the utilization stage, and selects the sparse scheme in the buffer that is the most forward, i.e. the highest return ; S5, adding perturbation to the model according to the structured coefficient scheme a perturbation scaling factor of 2 performing the perturbation, and calculating a first loss value ; S6, adding a structure coefficient scheme to the model a disturbance scaling factor of 2 performing a disturbance, calculating a second loss value ; S7, adding perturbations to the model according to the structured coefficient scheme a perturbation scaling factor of 2 performing perturbations to restore the model parameters; S8、according to the first loss value and the second loss value estimating the zeroth gradient ; S9, updating the model parameters using a zero-order stochastic gradient descent scheme; S10, maintaining and updating the sparse scheme and the reward buffer: using a sequence update function to update the buffer, wherein is a set of pruned layer information, is a reward score; the steps S3-S10 are repeated for a preset number of times to obtain a model with the minimum loss.

2. The method of claim 1, wherein, The natural language model is a large artificial intelligence neural network based on a Transformer structure, including a model based on an encoder structure and a model based on a decoder structure.

3. The method of claim 2, wherein, The model based on the encoder structure is composed of each encoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a feedforward network, a residual connection network, and a layer normalization processing module. The model based on the decoder structure is composed of Each decoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, an inter-attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network, and a layer normalization processing module.

4. The method of claim 1, wherein, The zero-order optimization method refers to a zero-order optimization method capable of fine-tuning a language model containing a large number of parameters; parameters that the model needs to update in the fine-tuning step are obtained a zero-order optimizer is used to estimate the gradient of the model, and the gradient and the set perturbation information are used to update the parameters of the model.

5. The method of claim 4, wherein, The SPSA method is used to estimate the zero-order gradient, and the zero-order stochastic gradient descent scheme is used to update the model parameters. where the implementation of the SPSA method is to obtain a normal distribution matrix consistent with the scale of the model parameters , the objective function in the fine-tuning process of the model , in the case of small batches of data , the first-order gradient is estimated by slightly perturbing the model parameters in a random direction, then performing two forward propagations to calculate the difference between the predicted values of the objective function wherein is a perturbation scaling factor; denotes transpose, denotes gradient sign; The zero-order stochastic gradient descent scheme is used to update the parameters of the model as follows: In the first Expression for updating model parameters using zeroth-order gradient: In the formula, is a set of time instance model parameters, is a set of time instance training data, is a learning rate.

6. The method of claim 1, wherein, The structured pruning scheme is used to adjust the forward propagation process, including: using the layer structure of the language model as the most basic pruning structure, calculating the forward propagation of the parameters of the layer without executing the pruning scheme, and adding a perturbation item to the parameter matrix of the layer by performing matrix addition operation.

7. The method of claim 1, wherein, The genetic algorithm and the random selection method are used to dynamically adjust the pruning structure, including: Use genetic algorithms to adjust the structure of the model in each training step, first in the front In the steps, the first generation of the race is obtained by a random infrastructure dropout strategy; After these steps, the race is updated using the genetic strategy.

8. The method of claim 1, wherein, The reinforcement learning strategy is used to decide to explore or execute the best pruning scheme, including: Using - a greedy algorithm, to perform a search at the first training step, generating a set of unexplored layer sparsity schemes, using a predetermined strategy, and using the best of the already explored sparsity schemes with a probability of ; since the training is a dynamic process, the reward of each sparsity scheme is updated dynamically in the greedy strategy.

9. The method of claim 8, wherein, The revenue of each sparse scheme is dynamically updated, including: Adopting the negative value of the loss on the training mini-batch data as the reward, gradually increasing the reward for adopting the specified structured pruning scheme as the training proceeds, in order to synchronize the rewards of all structured pruning schemes as the training process proceeds, a discount factor is introduced , multiplying the rewards of all schemes stored in the buffer by .

10. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one program, which is loaded and executed by the processor to implement the method of any one of claims 1-9.

Images