Language model dynamic structuring efficient fine tuning method and electronic equipment

Abstract

The invention discloses a dynamic structured efficient fine tuning method for a language model and electronic equipment. The method comprises the following steps: acquiring a natural language model needing to be finely tuned; acquiring a data set of a natural language downstream task, and designing a prompt statement for guiding the model to understand the downstream task; performing fine tuning on the natural language model by adopting the data set and the prompt statement; wherein the fine tuning process comprises the following steps of: updating parameters of the model by using a zero-order optimization method; adjusting a forward propagation process by using a structured pruning scheme; dynamically adjusting a pruning structure by using a genetic algorithm and a random selection method; and determining to explore or execute an optimal pruning scheme by using a reinforcement learning strategy. Structured pruning, reinforcement learning and genetic algorithms are introduced to solve the problems of high calculation overhead and long training time in the zero-order optimization process. According to the method, the calculation overhead of zero-order optimization on all models and tasks can be reduced on the premise of maintaining model performance, and the method can be widely applied to the field of natural language model fine adjustment.

Description

Technical Field

[0001] The present invention relates to the technical fields of text processing, pruning and zero-order optimization, and in particular to a language model dynamic structured efficient fine-tuning method and electronic equipment. Background Art

[0002] Language models excel in understanding and generating complex tasks and are increasingly being used. In order to quickly apply pre-trained language models to downstream tasks, fine-tuning techniques have become an efficient and common solution. However, as the parameter size and complexity of language models increase, the computational cost associated with fine-tuning rises rapidly. The large amount of resource requirements has created a very large obstacle to the application and research of language models. Therefore, designing a memory and computationally efficient fine-tuning method is of great significance for the low-cost application of language models.

[0003] In order to reduce the consumption of computing resources when fine-tuning large models, commonly used methods include model-level solutions and optimization-level solutions. At the model level, structured pruning or selective updating of some parameters are commonly used and effective methods. These solutions can reduce memory consumption by reducing the storage state of the first-order optimizer, but the first-order optimizer still requires a large amount of memory to store information such as gradients and momentum. At the optimizer level, the zero-order optimizer is proposed to replace the first-order optimizer. The zero-order optimizer uses approximate gradients and does not need to store the gradient information of each parameter, thus greatly saving memory consumption during model fine-tuning. However, since the zero-order gradient is obtained by approximate calculation through the differential algorithm, the convergence speed is very slow. How to design an optimization scheme that reduces both memory consumption and computing consumption during fine-tuning is a research direction worthy of in-depth exploration. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the object of the present invention is to provide a language model dynamic structured efficient fine-tuning method based on zero-order optimization, an electronic device and a medium.

[0005] The first technical solution adopted by the present invention is:

[0006] A language model dynamic structured efficient fine-tuning method, comprising the following steps:

[0007] Obtain the natural language model that needs fine-tuning; obtain the dataset of natural language downstream tasks and design prompts to guide the model to understand downstream tasks;

[0008] Fine-tune the natural language model using datasets and prompt sentences;

[0009] The fine-tuning process includes:

[0010] Update the model's parameters using a zero-order optimization method;

[0011] Use a structured pruning scheme to adjust the forward propagation process;

[0012] Use genetic algorithms and random selection methods to dynamically adjust the pruning structure;

[0013] Use reinforcement learning strategies to decide whether to explore or execute the best pruning solution;

[0014] In the test, the fine-tuned language model is used to obtain test results on downstream tasks.

[0015] Furthermore, the natural language model is a large artificial intelligence neural network based on the Transformer structure, including models based on the encoder structure (such as the BERT series models) and models based on the decoder structure (such as the Llama, OPT series models, etc.). The natural language model is a language model with certain language understanding capabilities.

[0016] Furthermore, the encoder structure-based model is composed of L cascaded encoder layers, each encoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module;

[0017] The model based on the decoder structure is composed of L cascaded decoder layers, each decoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module.

[0018] Furthermore, the data sets of the natural language downstream tasks include classification tasks, multiple-choice tasks and generation tasks, and the data consists of existing language texts.

[0019] Furthermore, the prompt sentence is: a targeted and suggestive sentence composed of texts in the same language as the downstream task dataset.

[0020] Furthermore, the zero-order optimization method refers to a zero-order optimization method capable of fine-tuning a language model containing a large number of parameters;

[0021] The parameters θ that need to be updated in the fine-tuning step are obtained, the gradient of the model is estimated using a zero-order optimizer, and the model parameters are updated using this gradient and the set perturbation information.

[0022] Among them, zero-order optimization is a parameter optimization method that does not require explicit calculation of the gradient information of the objective function.

[0023] Furthermore, the zero-order gradient is estimated using the SPSA method, and the model parameters are updated using the zero-order stochastic gradient descent (ZO-SGD) scheme;

[0024] The implementation scheme of the SPSA method is to obtain a normal distribution matrix z that is consistent with the scale of the model parameters, the objective function L in the fine-tuning process of the model, and in the case of a small batch of data B, by slightly perturbing the model parameters in random directions, and then performing two forward propagations to calculate the difference in the predicted values ​​of the objective function to estimate the first-order gradient:

[0025]

[0026] Where μ is the perturbation scaling factor; T represents transposition, Represents the gradient symbol;

[0027] The parameters of the model are updated using the zero-order stochastic gradient descent scheme as follows:

[0028] The expression for updating the model parameters using the zero-order gradient at the tth time is:

[0029]

[0030] In the formula, θ t is the model parameter set at time t, B t is the training data set at time t, and η is the learning rate.

[0031] Furthermore, the use of a structured pruning scheme to adjust the forward propagation process includes:

[0032] Use a structured sparse scheme to divide the model parameters, and use matrix operations to add perturbations to the model. For example, use the layer structure of the language model as the most basic pruning structure, perform forward propagation calculations on the parameters of the layer that does not implement the pruning scheme, and at the same time, perform matrix addition operations on the layer parameter matrix to add a perturbation term. Using modules or matrices as the basic structure of pruning is also an effective solution.

[0033] Furthermore, the dynamically adjusting the pruning structure using a genetic algorithm and a random selection method includes:

[0034] Use a genetic algorithm to adjust the structure of the model in each training step. First, in the first m steps, a random basic structure discarding strategy is used to obtain the first generation of the race. For example, the model layer is used as the basic structure, and the sequence number of the discarded layer each time constitutes the gene of each individual. After m+1 steps, the genetic strategy is used to update the race. For example, a simple genetic strategy is used to take all individuals in the race as parents to generate a new offspring. The genes of the offspring are generated based on the probability of the number distribution of all parent genes, and the number of occurrences of each gene is set to a minimum of 1 to maintain the mutation probability of the population, thereby maintaining an excellent population. This method is not limited to the use of the simple genetic algorithm mentioned above.

[0035] Furthermore, the use of a reinforcement learning strategy to determine the best pruning solution to explore or execute includes:

[0036] Use reinforcement learning schemes to balance the exploration and utilization of layer sparse solutions while maximizing the benefits of an efficient training process. For example, using an ∈-greedy algorithm, perform a search at the t-th training step with a probability of ∈, use a predetermined strategy to generate a set of unexplored layer sparse solutions, and use the best solution among the explored sparse solutions with a probability of 1-∈. Since training is a dynamic process, in the greedy strategy, the benefits of each sparse solution are dynamically updated.

[0037] Furthermore, the dynamically updating the benefit of each sparse scheme includes:

[0038] The negative value of the loss on the small batch data of training is used as the benefit. As the training progresses, the benefit of adopting the specified structured pruning scheme gradually increases. In order to synchronize the benefits of all structured pruning schemes as the training process progresses, a discount factor γ is introduced. In the next step, the benefits of all schemes stored in the buffer are multiplied by γ.

[0039] The second technical solution adopted by the present invention is:

[0040] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a language model dynamic structured efficient fine-tuning method as described above.

[0041] The third technical solution adopted by the present invention is:

[0042] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a language model dynamic structured efficient fine-tuning method as described above.

[0043] The fourth technical solution adopted by the present invention is:

[0044] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method.

[0045] The beneficial effects of the present invention are as follows: the present invention proposes an innovative, dynamic and efficient language model fine-tuning scheme, which can be applied to the fine-tuning of any pre-trained language model on any downstream task. In order to achieve memory efficiency, the present invention uses a zero-order optimization method to update the model parameters, which greatly saves the storage consumption of the parameter state. The use of a structured pruning scheme reduces the computational overhead in the fine-tuning process. At the same time, in order to reduce the hard errors caused by structured pruning, the present invention proposes a structure search strategy for dynamic pruning, which uses an ∈-greedy algorithm and a genetic algorithm to update the pruning structure. With the combination of the above schemes, the present invention realizes a memory and computationally efficient zero-order dynamic structured fine-tuning scheme, which can effectively reduce resource consumption in the language model fine-tuning process while ensuring the performance of fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 is a schematic diagram of a structured sparse solution in an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of dynamically updating the benefit of each sparse scheme in an embodiment of the present invention;

[0049] Figure 3 It is a greedy algorithm flow chart / genetic algorithm flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0051] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0052] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0053] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0054] Terminology explanation:

[0055] Language Model: A language model is a deep learning model that can process natural language. It learns the statistical laws and semantic representation of language by pre-training on a large-scale corpus. Large language models usually have high language understanding and generation capabilities and can be used for a variety of natural language processing tasks.

[0056] Zero-Order Optimizer: Zero-order optimization generally refers to optimization methods that do not require gradient information. Generally, it refers to a method based on parameter adoption and differential ideas to estimate the gradient of the objective function with respect to the parameter update direction. A zero-order optimizer refers to an optimizer that uses zero-order optimization methods to update model parameters.

[0057] First-Order Optimizer: A first-order optimizer is an optimizer that uses back-propagation and other strategies to calculate the gradient information and moment information of model parameters, and uses this information to update the model parameters.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment provides a dynamic structured efficient fine-tuning method for a language model based on zero-order optimization, the method comprising a model sparsification scheme that can be used for structure search, a reinforcement learning scheme for enhancing the benefits of the fine-tuning process, a genetic algorithm for sparse structure construction, a sparse zero-order optimization scheme, a sparse zero-order stochastic gradient descent scheme, and a zero-order structured fine-tuning training strategy. The model sparsification scheme that can be used for structure search can reduce floating-point calculations in the model optimization process and improve computational efficiency; reinforcement learning can effectively balance the exploration consumption of the sparse structure of the model and the consumption of the fine-tuning process during the fine-tuning process, reducing the performance degradation of the model relative to the ordinary zero-order optimization process; the genetic algorithm used for sparse structure construction can reduce unnecessary exploration in the structure search of reinforcement learning, narrow the exploration range, and improve the fine-tuning efficiency; the sparse zero-order optimization scheme can avoid the calculation of the gradient back propagation process, and compared with the first-order optimizer, it can reduce the storage of the optimizer state to achieve the purpose of saving memory overhead; the sparse zero-order stochastic gradient descent scheme can effectively adjust all parameters, while reducing the number of updated parameters and reducing the time consumption of fine-tuning; through the combination of the above schemes, a zero-order structured fine-tuning training strategy is obtained, which can greatly reduce the memory and time consumption in the language model fine-tuning process, and at the same time, it can obtain an effective structure pruning scheme for downstream tasks and obtain the importance distribution of model parameters. The method specifically includes the following steps:

[0060] S1. Obtain a dataset for a specified natural language downstream task, design a suitable prompt to construct data that is effective for zero-order fine-tuning, and obtain a natural language model that needs to be fine-tuned.

[0061] In this embodiment, the natural language downstream task dataset is selected as SST-2, and the natural language model that needs to be fine-tuned is selected as OPT-13B. For the SST-2 dataset, a suitable prompt is designed. Here, the prompt format is set to {text}It was{terrible or great}. The text information in the SST-2 dataset needs to be filled in at {text}, and the label information in the SST-2 dataset needs to be filled in at {terrible or great}. Use either terrible or great. For the OPT-13B model, it is a pre-trained decoder-based natural language model that has certain capabilities for understanding and expressing natural language. In Table 1, more prompt designs for other natural language datasets that can be used for zero-order optimization are provided.

[0062] Table 1 Prompt that can be used for other natural language downstream task datasets in the present invention

[0063]

[0064] S2. Initialize hyperparameters.

[0065] There are a lot of hyperparameters in this method. In this embodiment, the hyperparameters used are set as follows: the number of sparse layers n = 6; the set of layer information a is obtained by random initialization; the empty buffer sequence The size of m = 100; the initial exploration probability ∈ = 0.9; the minimum exploration probability β = 0.3; the perturbation scaling factor μ = 1e-3; the discount factor γ = 0.9; the decay factor α = 0.99; the total training steps T = 20000; the learning rate l = 1e-7; the size of the training data set is set to 1000; the batch data size B = 16.

[0066] S3,∈-The greedy strategy decides whether to explore or exploit.

[0067] The implementation scheme of the greedy strategy is to first generate a random number ρ between 0 and 1, determine the size of the random number ρ and ∈, and when ρ<∈, execute the exploration process, and when ρ>∈, execute the utilization process. As the exploration of the layer sparse structure deepens, layer importance information will exist in the buffer, so the exploration probability is gradually reduced in this embodiment. After each execution of the greedy strategy, the exploration probability will be updated to ∈=max(β,α∈), and ∈ will gradually decrease to the minimum exploration probability β during the exploration process, which is achieved in about 2000 loop steps. This setting can quickly explore the entire layer sparse structure space. Figure 1 The layer sparsification scheme and the reasons for the speedup are shown in detail in .

[0068] S4, genetic algorithm updates the model structured sparse solution. The buffer obtains the solution with the highest return.

[0069] exist Figure 2 The selection strategy of the structured sparse scheme is shown in . When the ∈-greedy algorithm selects the exploration phase, this embodiment uses a simplified genetic algorithm to update the sparse layer structure information a. The detailed process is as follows: when the information stored in the buffer is less than m, the buffer area is not full, and a layer sparse set a containing n elements is generated by random sampling; when the buffer is full, the occurrence probability of all layers in the buffer is counted, and n layer numbers are extracted from 1, ..., N without duplication according to this probability to form a. When the ∈-greedy algorithm selects the utilization phase, the sparse scheme a that is at the front of the buffer, that is, the one with the highest return, is selected. Figure 3 The layer sparsification strategy update scheme at each step is shown in detail in .

[0070] S5. Add +μ disturbance to the model according to the structured coefficient scheme and calculate l + .

[0071] In this embodiment, the target loss of the model during training is Where K is the total length of the sequence, P(x k |x1,x2,…,x k-1 ) is the probability of the kth word predicted by the model given the previous word. By calculating the negative log-likelihood of each word and minimizing this loss, the model's capacity is increased. When a sparse set of layers is obtained, first obtain the parameter set of the layer to be skipped based on a:

[0072]

[0073] Determine a random seed s, generate a random Gaussian perturbation of the same size as the adjustment parameter, and scale it with a scaling factor of +1:

[0074]

[0075]

[0076] S6. Add -2μ disturbance to the model according to the structured coefficient scheme and calculate l - .

[0077] Reset the random seed s, generate random Gaussian perturbations of the same size as the adjustment parameters, and scale them with a scaling factor of -2:

[0078]

[0079]

[0080] S7. Add +u disturbance to the model according to the structured coefficient scheme and restore the model parameters.

[0081] Reset the random seed s, generate random Gaussian perturbations of the same size as the adjustment parameters, and scale them with a scaling factor of +1:

[0082]

[0083] S8, estimate the zero-order gradient project_grad = (l + +l - ) / 2.

[0084] In this implementation, the SPSA method is used to estimate the first-order gradient. The difference estimation is performed based on the loss values ​​obtained after two disturbances to obtain the first-order estimated gradient:

[0085]

[0086] S9. Update the model parameters using the zero-order stochastic gradient descent (ZO-SGD) scheme.

[0087] Reset the random seed s, and update the sparse layer parameters using stochastic gradient descent according to the zero-order gradient project_grad obtained in step S8:

[0088]

[0089] S10. Maintain and update the sparse scheme and reward buffer.

[0090] The buffer is updated using the sequence update function UpdateSeq(a,r,project_grad), where a is the set of pruned layer information, r is the reward score, and project_grad is the zero-order estimated gradient:

[0091] right All of:

[0092]

[0093] Insert [a,r] to satisfy location.

[0094] Repeat steps S3-S10 20,000 times to obtain the model with the lowest loss on the test set in this process.

[0095] S11. Use the trained language model to obtain results on the downstream task test dataset.

[0096] In order to verify the effectiveness of the present invention, after training on the SST-2 training set, it was tested on the SST-2 test set, using accuracy as the evaluation criterion, that is, the proportion of correct classification on all test samples. In addition to the SST-2 data set, in order to further prove the effectiveness, this embodiment is also tested on other models and data sets of different sizes. And compared with other methods. The present invention improves the calculation speed by 10%+, and the performance is similar to that of the ordinary zero-order optimization method. Table 2 shows the performance of the present invention, where DROP uses the F1 score.

[0097] Table 2 Performance of the present invention using different downstream task datasets on the Llama-7B and OPT-13B models

[0098]

[0099]

[0100] In summary, compared with the prior art, this embodiment has the following advantages and beneficial effects:

[0101] (1) The method of this embodiment includes a dynamic structured sparse scheme, which effectively reduces the computational cost of zero-order optimization and reduces training time.

[0102] (2) The method of this embodiment also includes a sparse layer structure search scheme, which can search for layers that are relatively unimportant to model reasoning while maintaining the training effect during the training process.

[0103] (3) By adjusting the training process through the reinforcement learning algorithm, it is possible to balance the benefits of the training process and the benefits of layer structure exploration, and ultimately complete the training task while maintaining the training effect.

[0104] (4) The language model dynamic structured efficient fine-tuning method based on zero-order optimization proposed in this embodiment can be used in combination with other efficient fine-tuning (PEFT) schemes to further improve the training speed of zero-order optimization and has wide application value.

[0105] Example 2

[0106] A dynamic structured efficient fine-tuning method based on zero-order optimization includes the following steps:

[0107] Step 1: Obtain a natural language downstream task dataset , based on the task type, build a prompt and determine a target loss function .

[0108] Step 2: Get a pre-trained language model, get the number of layers N of the model, and all the parameters of the model are θ.

[0109] Step 3: Based on the dataset and prompt, fine-tune the language model and update the parameter θ. The fine-tuning process is as follows:

[0110] Set a set of learning rate schedules {η t}; Initialize a set a of layer information, which contains any n layer numbers from 1,…,N; Initialize an empty buffer sequence The size is m, where the element is a data pair consisting of a layer information set and a score; initialize an exploration probability 0≤∈<1. Define a minimum exploration probability 0≤β<1; define a perturbation scaling factor μ; define a discount factor γ; define a decay factor α.

[0111] 1) Define a perturbation function Perturb(θ,μ,s), where θ is a parameter set, μ is a scaling factor, and s is a random seed:

[0112] Set the global random seed to s;

[0113] For any θ i ∈θ:

[0114] Generate a random normally distributed perturbation

[0115] Update the parameter θ i =θ i +μz;

[0116] Returns the updated parameters θ.

[0117] 2) Define a sequence update function UpdateSeq(a,r,project_grad), where a is the set of pruned layer information, r is the reward score, and project_grad is the zero-order estimated gradient:

[0118]

[0119] 3) Processing the dataset All samples in the splice prompt;

[0120] For the training process of step t from 1 to T:

[0121] Get the parameter set of the layer to be skipped according to a

[0122] Randomly select b samples from the dataset to form a small training batch ;

[0123] Randomly initialize a random seed s;

[0124] Apply a random perturbation of size +μ to the model parameter subset θ′, and update the parameter subset θ′=Perturb(θ,μ,s);

[0125] Calculate the loss of the model at this time

[0126] Apply a random perturbation with a perturbation size of -2μ to the model parameter subset θ′, and update the parameter subset θ′=Perturb(θ,-2μ,s);

[0127] Calculate the loss of the model at this time

[0128] Apply a random perturbation with a perturbation size of -μ to the model parameter subset θ′, and update the parameter subset θ′=Perturb(θ,-2μ,s);

[0129] Compute the gradient using the difference estimate

[0130] Reset the global random seed to s;

[0131] Update the model parameters, for any θ i ∈θ:

[0132] Generate a random normally distributed perturbation

[0133] Update the parameter θ i =θ i -η×project_grad×z;

[0134] Updates the collection of layer information

[0135] Get the fine-tuned language model parameters θ.

[0136] In summary, this embodiment provides a dynamic structured efficient fine-tuning method for a language model based on zero-order optimization, which includes obtaining a data set of a specified natural language downstream task, designing a suitable prompt to construct data that is effective for zero-order fine-tuning, obtaining a natural language model that needs to be fine-tuned, initializing hyperparameters, using a ∈-greedy strategy to decide whether to explore or exploit, updating a model structured sparse scheme or a buffer using a genetic algorithm to obtain the scheme with the highest return, adding +μ disturbance to the model according to the structured coefficient scheme, and calculating l + ; Add -2μ disturbance to the model according to the structured coefficient scheme and calculate l -; Add +μ disturbance to the model according to the structured coefficient scheme to restore the model parameters; estimate the zero-order gradient project_grad = (l + +l - ) / 2μ; use ZO-SGD to update model parameters; maintain and update sparse schemes and reward buffers; loop steps; use the trained language model to obtain results on downstream task test data sets. The problems of high computational overhead and long training time in the zero-order optimization process are solved by introducing structured pruning, reinforcement learning and genetic algorithms. The method of this embodiment can reduce the computational overhead of zero-order optimization on all models and tasks while maintaining model performance, and can be widely used in the field of natural language model fine-tuning.

[0137] Example 3

[0138] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 The method shown.

[0139] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0140] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0141] Since the electronic device is an electronic device corresponding to a language model dynamic structured efficient fine-tuning method in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0142] Example 4

[0143] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 The method shown.

[0144] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0145] Since the storage medium is a storage medium corresponding to a language model dynamic structured efficient fine-tuning method in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0146] Example 5

[0147] In some possible implementations, various aspects of the method of the embodiments of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a language model dynamic structured efficient fine-tuning method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0148] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0150] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A dynamic structured and efficient fine-tuning method for a language model, characterized in that: The following steps are involved: Obtain the natural language model that needs fine-tuning; obtain the dataset of natural language downstream tasks and design prompt sentences to guide the model to understand downstream tasks; Fine-tune the natural language model using datasets and prompt sentences; The fine-tuning process includes: Update the model's parameters using a zero-order optimization method; Use a structured pruning scheme to adjust the forward propagation process; Use genetic algorithms and random selection methods to dynamically adjust the pruning structure; Use reinforcement learning strategies to decide whether to explore or execute the best pruning solution.

2. A language model dynamic structured efficient fine-tuning method according to claim 1, characterized in that: The natural language model is a large artificial intelligence neural network based on a Transformer structure, including a model based on an encoder structure and a model based on a decoder structure.

3. A language model dynamic structured efficient fine-tuning method according to claim 2, characterized in that: The encoder structure-based model is composed of L cascaded encoder layers, each encoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module; The model based on the decoder structure is composed of L cascaded decoder layers, each decoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module.

4. The method for dynamic structural efficient fine-tuning of a language model according to claim 1, characterized in that: The zero-order optimization method refers to a zero-order optimization method capable of fine-tuning a language model containing a large number of parameters; The parameters θ that need to be updated in the fine-tuning step are obtained, the gradient of the model is estimated using a zero-order optimizer, and the model parameters are updated using this gradient and the set perturbation information.

5. A language model dynamic structured efficient fine-tuning method according to claim 4, characterized in that: Use the SPSA method to estimate the zero-order gradient and use the zero-order stochastic gradient descent scheme to update the model parameters; The implementation of the SPSA method is to obtain a normal distribution matrix z that is consistent with the scale of the model parameters. The objective function in the fine-tuning process of the model is In the case of a small batch of data B, the first-order gradient is estimated by slightly perturbing the model parameters in random directions and then calculating the difference between the predicted values ​​of the objective function after two forward propagations: Where μ is the perturbation scaling factor; T represents transposition, Represents the gradient symbol; The parameters of the model are updated using the zero-order stochastic gradient descent scheme as follows: The expression for updating the model parameters using the zero-order gradient at the tth time is: In the formula, θ t is the model parameter set at time t, B t is the training data set at time t, and η is the learning rate.

6. The method for dynamic structural efficient fine-tuning of a language model according to claim 1, characterized in that: The structured pruning scheme is used to adjust the forward propagation process, including: The layer structure of the language model is used as the most basic pruning structure. The parameters of the layer that does not implement the pruning scheme are forward propagated and calculated. At the same time, a matrix addition operation is performed on the parameter matrix of the layer to add a perturbation term.

7. The method for dynamic structural efficient fine-tuning of a language model according to claim 1, characterized in that: The method of dynamically adjusting the pruning structure using a genetic algorithm and a random selection method includes: A genetic algorithm is used to adjust the structure of the model in each training step. First, in the first m steps, a random base structure discarding strategy is used to obtain the first generation of the race; after m+1 steps, the genetic strategy is used to update the race.

8. The method for dynamic structural efficient fine-tuning of a language model according to claim 1, characterized in that: The use of reinforcement learning strategies to determine the best pruning solution to explore or execute includes: Using the ∈-greedy algorithm, the search is performed at the tth training step with probability ∈, a predetermined strategy is adopted to generate a set of unexplored layer sparse solutions, and the best solution among the explored sparse solutions is used with probability 1-∈; since training is a dynamic process, in the greedy strategy, the benefit of each sparse solution is dynamically updated.

9. A language model dynamic structured efficient fine-tuning method according to claim 8, characterized in that: The dynamically updating benefit of each sparse solution includes: The negative value of the loss on the small batch data of training is used as the benefit. As the training progresses, the benefit of adopting the specified structured pruning scheme gradually increases. In order to synchronize the benefits of all structured pruning schemes as the training process progresses, a discount factor γ is introduced. In the next step, the benefits of all schemes stored in the buffer are multiplied by γ.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 9.

Images