Method for finely adjusting large language model and electronic equipment

Abstract

The invention provides a method for finely adjusting a large language model and electronic equipment. Relates to the technical field of computers. The method comprises the following steps: in an ith fine tuning stage, sampling a parameter space of a large language model to obtain a model parameter vector; calculating the differential privacy zero-order gradient of the stage loss function based on the model parameter vector and the zero-order scale parameter; the stage loss function is obtained by updating a loss function of the large language model; based on the differential privacy zero-order gradient and the learning rate, performing iteration processing on the model parameters of the large language model according to a preset iteration round to obtain target model parameters of the ith fine tuning stage; according to the embodiment of the invention, the resource consumption of the fine tuning model can be reduced, and the model performance is ensured while the data privacy is effectively protected.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for fine-tuning a large language model and an electronic device. Background Art

[0002] A large language model (LLM) is a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Based on the pre-trained large language model, the model parameters are slightly adjusted to meet the needs of specific tasks. In the pre-training process of the large language model, large-scale unlabeled text datasets are usually used for training to learn the inherent structure and rules of the language; while the fine-tuning process uses labeled task-specific datasets to enable the large language model to achieve better performance on specific tasks.

[0003] The fine-tuning process of a large language model based on a specific task will involve private or sensitive training data. Attackers may infer the training data used for fine-tuning through model inversion attacks or gradient inference attacks. In order to prevent data leakage and not infringe on data privacy, differential privacy technology is currently used to inject noise into the gradient to fine-tune the LLM. However, as the scale of LLM continues to increase, the process of fine-tuning LLM generates huge resource consumption and time overhead, and excessive noise injection reduces model performance. Summary of the invention

[0004] According to various embodiments of the present application, a method and electronic device for fine-tuning a large language model are provided, which can reduce the resource consumption of the fine-tuning model, effectively protect data privacy while ensuring model performance.

[0005] In a first aspect, the present application provides a method for fine-tuning a large language model, the method comprising: In the i-th fine-tuning stage, the parameter space of the large language model is sampled to obtain a model parameter vector; based on the model parameter vector and the zero-order scale parameter, the differential privacy zero-order gradient of the stage loss function is calculated; the stage loss function is updated by the loss function of the large language model; based on the differential privacy zero-order gradient and the learning rate, the model parameters of the large language model are iteratively processed according to the preset iteration rounds to obtain the target model parameters of the i-th fine-tuning stage; wherein, the target model parameters of the i-th fine-tuning stage are used to construct the parameter space of the i+1-th fine-tuning stage, the iteration rounds of the i+1-th fine-tuning stage are greater than the iteration rounds of the i-th fine-tuning stage, the zero-order scale parameter of the i-th fine-tuning stage is less than the zero-order scale parameter of the i+1-th fine-tuning stage, the learning rate of the i-th fine-tuning stage is greater than the learning rate of the i+1-th fine-tuning stage, and i is an integer greater than or equal to 1.

[0006] Through the above method, the large language model is fine-tuned by dividing it into multiple fine-tuning stages, and the loss function and zero-order scale parameters are adjusted and updated according to the fine-tuning stage, so that the fine-tuning process of the large language model converges more smoothly and quickly, reducing the resource consumption and time overhead of fine-tuning the large language model; by sampling model parameter vectors of different dimensions and calculating the differential privacy zero-order gradient of the stage loss function, the impact of differential noise injection on model performance is reduced, while ensuring data privacy, the performance of the model is retained to the greatest extent; it has strong ease of use and practicality.

[0007] In a possible implementation manner of the first aspect, in the i-th fine-tuning stage, sampling the parameter space of the large language model to obtain a model parameter vector includes: In the i-th fine-tuning stage, the parameter space of the language model is sampled based on a fixed pruning rate, a dynamic pruning rate, or an incremental pruning rate to obtain the model parameter vector.

[0008] In a possible implementation manner of the first aspect, calculating a differentially private zero-order gradient of a stage loss function based on the model parameter vector and the zero-order scale parameter includes: Based on the model parameter vector and the zero-order scale parameter, the differentially private zero-order gradient of the stage loss function is calculated through the zero-order gradient estimation algorithm with inserted noise terms.

[0009] In a possible implementation manner of the first aspect, an expression of a zero-order gradient estimation algorithm with a noise term inserted is:

[0010] in, is the differential privacy zero-order gradient function, is the stage loss function, θ is the model parameter, β is the zero-order scale parameter, v is the model parameter vector, and noise is a Gaussian distribution. The noise term.

[0011] In a possible implementation manner of the first aspect, before calculating the differentially private zero-order gradient of the phase loss function, the method further includes: In the i-th fine-tuning stage, the loss function of the large language model is updated based on the regularization term to obtain the stage loss function; Among them, the expression of the loss function of updating the large language model based on the regularization term is:

[0012] is the stage loss function, is the loss function, are the model parameters before fine-tuning at the i-th fine-tuning stage, is the regularization term, and λ is the regularization constant.

[0013] In a possible implementation manner of the first aspect, before calculating the differentially private zero-order gradient of the phase loss function, the method further includes: Update the zero-order scale parameters based on the changes in the fine-tuning phase; Among them, the expression for updating the zero-order scale parameter is:

[0014] is the zero-order scale parameter of the i+1th fine-tuning stage; is the zero-order scale parameter of the i-th fine-tuning stage; k is the update coefficient, which is greater than 1.

[0015] In a possible implementation manner of the first aspect, before calculating the differentially private zero-order gradient of the phase loss function, the method further includes: Update the learning rate based on the changes during the fine-tuning phase; Among them, the expression for updating the learning rate is:

[0016] is the learning rate of the i+1th fine-tuning stage; is the learning rate of the i-th fine-tuning stage.

[0017] In a possible implementation manner of the first aspect, based on the differential privacy zero-order gradient and the learning rate, the model parameters of the large language model are iteratively processed according to preset iteration rounds to obtain the target model parameters of the i-th fine-tuning stage, including: Calculate the differentially private zero-order gradient of the stage loss function according to the preset iteration rounds; Based on the differentially private zero-order gradient calculated in each iteration round and the learning rate of the i-th fine-tuning stage, the model parameters of the large language model are updated to obtain the target model parameters of the i-th fine-tuning stage.

[0018] In a possible implementation manner of the first aspect, based on the differentially private zero-order gradient calculated in each iteration round and the learning rate of the i-th fine-tuning stage, an expression for updating the model parameters of the large language model is:

[0019] in, is the model parameter of the tth iteration of the i-th fine-tuning phase, is the model parameter of the t-1th iteration of the i-th fine-tuning phase, is the learning rate of the i-th fine-tuning stage, is the differentially private zero-order gradient corresponding to the stage loss function of the t-1th iteration round, and t is an integer greater than 1.

[0020] In a second aspect, the present application provides a device for fine-tuning a large language model, comprising: An acquisition unit, used to sample the parameter space of the large language model in the i-th fine-tuning stage to obtain a model parameter vector; A computing unit, configured to compute a differentially private zero-order gradient of a stage loss function based on the model parameter vector and a zero-order scale parameter; the stage loss function is obtained by updating the loss function of the large language model; An iteration unit is used to iteratively process the model parameters of the large language model according to a preset iteration round based on the differential privacy zero-order gradient and the learning rate, so as to obtain the target model parameters of the i-th fine-tuning stage; wherein the target model parameters of the i-th fine-tuning stage are used to construct the parameter space of the i+1-th fine-tuning stage, the iteration round of the i+1-th fine-tuning stage is greater than the iteration round of the i-th fine-tuning stage, the zero-order scale parameter of the i-th fine-tuning stage is less than the zero-order scale parameter of the i+1-th fine-tuning stage, the learning rate of the i-th fine-tuning stage is greater than the learning rate of the i+1-th fine-tuning stage, and i is an integer greater than or equal to 1.

[0021] In a third aspect, the present application provides an electronic device, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute any one of the methods described in the first aspect.

[0022] In a fourth aspect, the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method described in the first aspect.

[0023] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0024] In a sixth aspect, the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the method described in the first aspect above.

[0025] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a flow chart of a method for fine-tuning a large language model provided in an embodiment of the present application; Figure 2 A schematic diagram of the stages of model fine-tuning provided in an embodiment of the present application; Figure 3 A schematic diagram of the parameter space provided for an embodiment of the present application; Figure 4 A schematic diagram of a pruning strategy provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an apparatus for fine-tuning a large language model provided in an embodiment of the present application; Figure 6 A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0030] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0033] At present, by fine-tuning the pre-trained large language model, the large language model is more suitable for specific task requirements. Attackers may infer the training data used in the fine-tuning of the large language model through model inversion attacks or gradient inference attacks. In order to prevent data leakage, differential privacy stochastic gradient descent (DP-SGD) can be used to add noise to the gradient update to achieve first-order optimization based on differential privacy in the fine-tuning process.

[0034] However, as the scale of large language models continues to increase and the number of parameters continues to increase, DP-SGD's method of calculating gradients through backpropagation will bring about huge consumption of computing resources and time overhead; and when excessive noise is introduced, the performance of large language models will be reduced.

[0035] In response to the above problems, an embodiment of the present application provides a method for fine-tuning a large language model. Based on the zero-order gradient algorithm, differential privacy is added, and model parameters are sampled through a multi-stage pruning strategy to achieve effective protection of privacy data in the multi-stage fine-tuning process. Gradient estimation is performed based on the forward propagation calculation of the zero-order gradient algorithm to reduce storage overhead and gradient calculation time overhead. At the same time, appropriate noise injection is selected for different fine-tuning stages to ensure model performance.

[0036] The specific implementation process of this application is introduced below through embodiments.

[0037] See also Figure 1 , Figure 1 A flow chart of a method for fine-tuning a large language model provided in an embodiment of the present application, the method may include the following steps: S101, in the i-th fine-tuning stage, the parameter space of the large language model is sampled to obtain a model parameter vector.

[0038] In the embodiment of the present application, the fine-tuning process of the large language model may include multiple fine-tuning stages, and the i-th fine-tuning stage represents any stage in the fine-tuning process. In the fine-tuning process, the division of the fine-tuning stages can be set based on the actual application scenario requirements, for example, it can be divided into 3 or 5 fine-tuning stages, or more fine-tuning stages can be divided based on needs.

[0039] Exemplarily, at each fine-tuning stage, the parameter space is constructed by calculating the importance of the model parameters to the fine-tuning process. Figure 3 The parameter space shown contains three dimensions, each box can correspond to a model parameter, each layer corresponds to a dimension of the parameter space, such as the first dimension, the second dimension and the third dimension; each dimension is divided according to different standards. The parameter space can include model parameters of three dimensions, and each model parameter of each dimension corresponds to a different degree of importance for the current fine-tuning stage. Based on the importance of each model parameter, sampling is performed in one of the dimensions of the parameter space, and the model parameters whose importance meets the importance threshold are selected to form a model parameter vector.

[0040] Exemplarily, the importance of each model parameter can be calculated by a zero-order gradient estimation algorithm, and the calculated zero-order gradient represents the importance of the model parameter. For example, the larger the calculated zero-order gradient value is, the greater the importance of the model parameter is, and the smaller the zero-order gradient value is, the smaller the importance of the model parameter is. Accordingly, by establishing a conversion relationship between the importance and the value of the zero-order gradient, a score corresponding to the importance is calculated based on the zero-order gradient value. The importance of the model parameter can be expressed in percentage.

[0041] Exemplarily, by sampling the parameter space, a model parameter whose importance satisfies an importance threshold is selected in one dimension; Figure 2 As shown, in stage 1, the parameter space V 1 Sampling is performed to obtain the model parameter vector v. In stage i, the parameter space V i Sampling is performed to obtain the model parameter vector v. Figure 3 The model parameter vector is obtained based on the model parameter selected from one of the dimensions. That is, the embodiment of the present application does not fine-tune all model parameters of the large language model, but selects model parameters that meet the importance threshold at different fine-tuning stages and performs iterative updates at the current fine-tuning stage.

[0042] Among them, after the current fine-tuning stage ends, the parameter space of the next fine-tuning stage is constructed based on the model parameters output after iterative update in the fine-tuning stage; that is, based on the target model parameters in the i-th fine-tuning stage, the parameter space of the i+1-th fine-tuning stage is constructed. The process implementation principle of constructing the parameter space of the i+1-th fine-tuning stage is the same as the above method. In different fine-tuning stages, the order of the model parameters in the parameter space can be dynamically adjusted based on their importance.

[0043] Accordingly, in the next fine-tuning stage, the new parameter space is sampled to obtain the model parameter vector of the fine-tuning stage. For example, in the first fine-tuning stage, the first dimension of the parameter space is sampled to obtain the model parameter vector in the first dimension. In the second fine-tuning stage, by constructing a new parameter space, the second dimension of the new parameter space can be sampled to obtain the model parameter vector in the second dimension. By adjusting the parameter space and the model parameter vector obtained by sampling at each stage, it can be ensured that the direction sampling in different fine-tuning stages is more representative and diverse.

[0044] In some embodiments, in the i-th fine-tuning stage, the parameter space of the large language model is sampled to obtain a model parameter vector, including: In the i-th fine-tuning stage, the parameter space of the language model is sampled based on a fixed pruning rate, a dynamic pruning rate, or an incremental pruning rate to obtain a model parameter vector.

[0045] For example, during the fine-tuning process, the process of sampling the parameter space can be implemented based on a fixed pruning rate, a dynamic pruning rate, or an incremental pruning rate. Figure 4 As shown, all boxes are used to represent all model parameters of the large language model. Each box represents a model parameter set, which includes a certain number of model parameters. The model parameters selected in each fine-tuning stage constitute a model parameter vector.

[0046] Among them, Figure 4 As shown in Figure (a) in , the fixed pruning rate is to fine-tune different model parameters in each fine-tuning stage, and the number of selected model parameters in each fine-tuning stage is the same. Figure 4 As shown in Figure (b) in Figure 2, the dynamic pruning rate is that as the fine-tuning stage increases, the number of selected model parameters also increases dynamically. Figure 4 As shown in Figure (c) in the figure, the incremental pruning rate is that as the number of fine-tuning stages increases, the number of selected model parameters continues to increase, and the model parameters fine-tuned in each fine-tuning stage include not only the newly selected parameters in the current fine-tuning stage, but also the model parameters that have been fine-tuned in the previous fine-tuning stage.

[0047] Exemplarily, sampling is performed based on a fixed pruning rate, and the number of selected model parameters in each fine-tuning stage is fixed. As the large language model is fine-tuned, the contribution of different model parameters in the large language model to the optimization target changes. In each fine-tuning stage, based on the training status of the large language model and the optimization target, a set of model parameters are reselected for fine-tuning based on importance. By updating the set of model parameters that need to be fine-tuned in real time, it is ensured that the fine-tuned parameters are always the most representative parameters in the current stage, and changes in the large language model can be flexibly responded to to ensure the optimization results of each fine-tuning stage.

[0048] For example, sampling based on a dynamic pruning rate is more flexible than sampling with a fixed pruning rate. As the large language model is gradually optimized in each fine-tuning stage, the number of model parameters involved in fine-tuning in the fine-tuning stage also increases, so that more model parameters can be adjusted, and the potential of the large language model can be further optimized. The large language model can be concentrated on optimizing a small number of key parameters in the early fine-tuning stage, and more model parameters can be fully optimized in the later fine-tuning stage, so that the large language model can be more fully optimized, thereby improving the performance of the large language model as a whole.

[0049] For example, sampling based on incremental pruning rate makes the sampling process more flexible; it can more comprehensively optimize the performance of the large language model, enhance the robustness of the fine-tuning process, and ensure that the large language model maintains a high degree of stability in the expanding parameter space. The incremental pruning rate retains the model parameters fine-tuned in the previous stage, making the optimization of the large language model more continuous and consistent; and further optimizes based on the optimization results of each fine-tuning stage, continuously accumulates optimization results, and significantly improves the performance of the large language model; it is suitable for high-precision and high-stability task scenarios.

[0050] The above sampling process of the parameter space is the process of selecting the model parameters that need to be fine-tuned in each fine-tuning stage, that is, determining the fine-tuning object in each fine-tuning stage.

[0051] The embodiment of the present application reduces the estimation error of the subsequent differential privacy zero-order gradient estimation and the scale of the introduced differential privacy noise through the sampling method of the staged pruning rate.

[0052] In some embodiments, the method further includes: in the i-th fine-tuning stage, updating the loss function of the large language model based on the regularization term to obtain a stage loss function; wherein the expression of the loss function of the large language model updated based on the regularization term is:

[0053] is the stage loss function, is the loss function, are the model parameters before fine-tuning at the i-th fine-tuning stage, is the regularization term, and λ is the regularization constant.

[0054] Exemplarily, at each fine-tuning stage, the loss function of the large language model is updated to obtain a stage loss function corresponding to each fine-tuning stage; Figure 2 The stage loss function corresponding to stage 1 shown , the stage loss function corresponding to stage i ; Among them, the model parameters corresponding to stage 1 are are the model parameters before fine-tuning, is the model parameter obtained after fine-tuning in the i-1th stage, and the regularization constant λ is used to determine the weight of the regularization term in the loss function.

[0055] By adding a regularization term to the loss function of the large language model, the volatility of the loss function in different fine-tuning stages is reduced, the model parameters or the output of the large language model in two consecutive fine-tuning stages are constrained, the change in the model parameters or the output of the large language model between the two fine-tuning stages is reduced, and the stability of the large language model is improved; the large language model is continuously approached to the target optimization direction, and the problem of overfitting or insufficient adjustment caused by one-time adjustment can be avoided.

[0056] In some embodiments, the method further includes: updating the zero-order scale parameter based on the change in the fine-tuning phase; the expression for updating the zero-order scale parameter is:

[0057] in, is the zero-order scale parameter of the i+1th fine-tuning stage; is the zero-order scale parameter of the i-th fine-tuning stage; k is the update coefficient, which is greater than 1.

[0058] Exemplarily, based on the increase in the fine-tuning stage, the zero-order scale parameter is updated, and the value of the zero-order scale parameter is continuously increased, so that in the later fine-tuning stage, the estimation of the differential privacy zero-order gradient is more stable. Figure 2 As shown, the zero-order scale parameter used in stage 1 corresponds to , corresponding to the zero-order scale parameter used in stage i .

[0059] In some embodiments, the method further includes: updating the learning rate based on the changes in the fine-tuning phase; the expression for updating the learning rate is:

[0060] in, is the learning rate of the i+1th fine-tuning stage; is the learning rate of the i-th fine-tuning stage.

[0061] For example, based on the increase of the fine-tuning stage, the learning rate is updated and the value of the learning rate is continuously reduced; based on the gradually decreasing learning rate, the large language model can adjust the model parameters to a greater extent in the early fine-tuning stage, and make more subtle adjustments in the later fine-tuning stage, thereby improving the convergence effect of the large language model fine-tuning process. Figure 2 As shown, the learning rate used in stage 1 is , the learning rate used in stage i .

[0062] S102, based on the model parameter vector and the zero-order scale parameter, calculate the differentially private zero-order gradient of the stage loss function; the stage loss function is updated by the large language model loss function.

[0063] In an embodiment of the present application, the differential privacy zero-order gradient is the gradient of the stage loss function with respect to the model parameters, which is obtained by predicting the difference between the two stage loss functions; the zero-order scale parameter is used to determine the distance between the two stage loss functions.

[0064] Exemplarily, by sampling near the current value of the model parameter, the value of the stage loss function of these sampling points is calculated; by comparing the difference between the value of the stage loss function of the sampling point and the value of the stage loss function corresponding to the original model parameter, the gradient of the stage loss function with respect to the model parameter is estimated, and the gradient indicates the direction of the fastest descent of the stage loss function in the parameter space. By using forward propagation to calculate the value of the stage loss function, the differential privacy zero-order gradient is estimated, avoiding the computational complexity of back propagation, and reducing computational overhead and storage requirements.

[0065] In some embodiments, based on the model parameter vector and the zero-order scale parameter, calculating the differentially private zero-order gradient of the stage loss function includes: Based on the model parameter vector and the zero-order scale parameter, the differentially private zero-order gradient of the stage loss function is calculated by inserting the zero-order gradient estimation algorithm with noise term. The expression of the zero-order gradient estimation algorithm with noise term is:

[0066] in, is the differential privacy zero-order gradient function, is the stage loss function, θ is the model parameter, β is the zero-order scale parameter, v is the model parameter vector, and noise is a Gaussian distribution. The noise term.

[0067] Exemplarily, noise is added to the gradient estimation so that differential privacy is guaranteed during the gradient estimation process to prevent sensitive information in the training data from being leaked from the gradient information. The noise term is a value of a noise parameter sampled from a Gaussian distribution.

[0068] S103, based on the differential privacy zero-order gradient and learning rate, iteratively process the model parameters of the large language model according to a preset number of iterations to obtain the target model parameters of the i-th fine-tuning stage.

[0069] In the embodiments of the present application, Figure 2 As shown, different iteration rounds are set for different fine-tuning stages, such as the iteration round T corresponding to stage 1. 1 , the number of iterations corresponding to stage i is T i ; As the fine-tuning stage increases, the number of iterations increases; for example, T i+1 =2*T i .

[0070] In some embodiments, based on the differential privacy zero-order gradient and the learning rate, the model parameters of the large language model are iteratively processed according to a preset iteration round to obtain the target model parameters of the i-th fine-tuning stage, including: According to the preset iteration rounds, the differential privacy zero-order gradient of the stage loss function is calculated; based on the differential privacy zero-order gradient calculated in each iteration round and the learning rate of the i-th fine-tuning stage, the model parameters of the large language model are updated to obtain the target model parameters of the i-th fine-tuning stage.

[0071] Exemplarily, during the iterative processing of model parameters, based on the value of each model parameter, the value of the differentially private zero-order gradient of the stage loss function corresponding to the parameter is calculated, and based on the zero-order gradient of the current round and the value of the model parameter of the current round, the new value corresponding to the model parameter is calculated, and the new value is the value of the model parameter after one iteration round; the new value of the model parameter is used as the input value of the next iteration round, and the differentially private zero-order gradient of the stage loss function with respect to the model parameter is calculated, and based on the differentially private zero-order gradient and the learning rate, the new value of the model parameter is continued to be calculated, thereby iterating the value of the model parameter in a loop to obtain the final value of the model parameter, that is, the target model parameter of the fine-tuning stage.

[0072] In the above-mentioned iterative process, the updated object may be a set of values ​​corresponding to a set of model parameters, and the values ​​of a set of model parameters are iteratively updated.

[0073] In some embodiments, based on the differentially private zero-order gradient calculated in each iteration round and the learning rate of the i-th fine-tuning stage, the expression for updating the model parameters of the large language model is:

[0074] in, is the model parameter of the tth iteration of the i-th fine-tuning stage; are the model parameters of the t-1th iteration of the i-th fine-tuning stage; is the learning rate of the i-th fine-tuning stage; is the differentially private zero-order gradient corresponding to the stage loss function of the t-1th iteration round; t is an integer greater than 1, and the value range can be 1 to T i An integer.

[0075] By dividing the fine-tuning process, the parameter space is sampled in different directions based on multiple fine-tuning stages to improve sampling diversity; noise is added during the gradient estimation process to ensure differential privacy; and multiple rounds of iterative updates are performed in each fine-tuning stage. Through the embodiments of the present application, the parameter space is sampled in different dimensions, the changes in the differential privacy zero-order gradient in different dimensions are calculated, and based on the changes in different dimensions, noise is distributed in the gradient estimation process, and noise items of different scales are added to avoid unnecessary noise interference and reduce the negative impact of noise on the large language model. While ensuring the privacy of training data, the performance of the large language model is retained to the maximum extent. By dividing the fine-tuning process, based on multiple fine-tuning stages, the stage loss function and zero-order scale parameters, learning rate, iteration rounds and other hyperparameters are adjusted respectively, so that the large language model can converge more smoothly and quickly, avoiding large-scale oscillations or overfitting problems. In the process of sampling the parameter space, some parameters of the large language model are sampled and adjusted based on different pruning rates, which reduces unnecessary parameter updates, reduces the dimension and estimation error of the differential privacy zero-order gradient estimation, and reduces the interference of irrelevant model parameters, making the fine-tuning process of the large language model more accurate and with smaller errors, thereby improving the overall performance of the large language model. At the same time, the sampling method based on the pruning rate reduces the scale of noise that needs to be added to ensure privacy protection, reduces the computational complexity, and reduces the impact of noise on the performance of the large language model.

[0076] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0077] Corresponding to the method of fine-tuning the large language model provided in the above embodiment, Figure 5 As shown, an embodiment of the present application provides a device for fine-tuning a large language model. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0078] The device includes: An acquisition unit 51 is used to sample the parameter space of the large language model in the i-th fine-tuning stage to obtain a model parameter vector; A calculation unit 52, configured to calculate a differentially private zero-order gradient of a stage loss function based on the model parameter vector and the zero-order scale parameter; the stage loss function is obtained by updating the loss function of the large language model; An iterative unit 53 is used to iteratively process the model parameters of the large language model according to a preset number of iterations based on the differential privacy zero-order gradient and the learning rate, to obtain the target model parameters of the i-th fine-tuning stage; wherein the target model parameters of the i-th fine-tuning stage are used to construct the parameter space of the i+1-th fine-tuning stage, the iterative rounds of the i+1-th fine-tuning stage are greater than the iterative rounds of the i-th fine-tuning stage, the zero-order scale parameter of the i-th fine-tuning stage is less than the zero-order scale parameter of the i+1-th fine-tuning stage, the learning rate of the i-th fine-tuning stage is greater than the learning rate of the i+1-th fine-tuning stage, and i is an integer greater than or equal to 1.

[0079] In a possible implementation, the acquisition unit 51 is further configured to sample the parameter space of the language model based on a fixed pruning rate, a dynamic pruning rate, or an incremental pruning rate in the i-th fine-tuning stage to obtain the model parameter vector.

[0080] In a possible implementation, the computing unit 52 is further configured to compute the differentially private zero-order gradient of the stage loss function based on the model parameter vector and the zero-order scale parameter by inserting a noise term into the zero-order gradient estimation algorithm.

[0081] In one possible implementation, the expression of the zero-order gradient estimation algorithm with the noise term inserted is:

[0082] in, is the differential privacy zero-order gradient function, is the stage loss function, θ is the model parameter, β is the zero-order scale parameter, v is the model parameter vector, and noise is a Gaussian distribution. The noise term.

[0083] In a possible implementation, the computing unit 52 is further configured to update the loss function of the large language model based on the regularization term in the i-th fine-tuning stage to obtain a stage loss function; Among them, the expression of the loss function of updating the large language model based on the regularization term is:

[0084] is the stage loss function, is the loss function, are the model parameters before fine-tuning at the i-th fine-tuning stage, is the regularization term, and λ is the regularization constant.

[0085] In a possible implementation, the calculation unit 52 is further configured to update the zero-order scale parameter based on the change in the fine-tuning phase; wherein the expression for updating the zero-order scale parameter is:

[0086] is the zero-order scale parameter of the i+1th fine-tuning stage; is the zero-order scale parameter of the i-th fine-tuning stage; k is the update coefficient, which is greater than 1.

[0087] In a possible implementation, the calculation unit 52 is further configured to update the learning rate based on the changes in the fine-tuning phase; wherein the expression for updating the learning rate is:

[0088] is the learning rate of the i+1th fine-tuning stage; is the learning rate of the i-th fine-tuning stage.

[0089] In one possible implementation, the iteration unit 53 is also used to calculate the differential privacy zero-order gradient of the stage loss function according to a preset iteration round; based on the differential privacy zero-order gradient calculated in each iteration round and the learning rate of the i-th fine-tuning stage, the model parameters of the large language model are updated to obtain the target model parameters of the i-th fine-tuning stage.

[0090] In one possible implementation, based on the differentially private zero-order gradient calculated in each iteration and the learning rate of the i-th fine-tuning stage, the expression for updating the model parameters of the large language model is:

[0091] in, is the model parameter of the tth iteration of the i-th fine-tuning phase, is the model parameter of the t-1th iteration of the i-th fine-tuning phase, is the learning rate of the i-th fine-tuning stage, is the differentially private zero-order gradient corresponding to the stage loss function of the t-1th iteration round, and t is an integer greater than 1.

[0092] Through the embodiments of the present application, the parameter space is sampled in different dimensions, the changes in the differential privacy zero-order gradient in different dimensions are calculated, and based on the changes in different dimensions, noise is distributed in the gradient estimation process, and noise items of different scales are added to avoid unnecessary noise interference and reduce the negative impact of noise on the large language model. While ensuring the privacy of training data, the performance of the large language model is retained to the maximum extent. By dividing the fine-tuning process, based on multiple fine-tuning stages, the stage loss function and zero-order scale parameters, learning rate, iteration rounds and other hyperparameters are adjusted respectively, so that the large language model can converge more smoothly and quickly, avoiding large-scale oscillations or overfitting problems. In the process of sampling the parameter space, some parameters of the large language model are sampled and adjusted based on different pruning rates, which reduces unnecessary parameter updates, reduces the dimension and estimation error of the differential privacy zero-order gradient estimation, and reduces the interference of irrelevant model parameters, making the fine-tuning process of the large language model more accurate and with smaller errors, thereby improving the overall performance of the large language model. At the same time, the sampling method based on the pruning rate reduces the scale of noise that needs to be added to ensure privacy protection, reduces the computational complexity, and reduces the impact of noise on the performance of the large language model.

[0093] Figure 6 A schematic diagram of the hardware structure of the electronic device 6 is shown.

[0094] like Figure 6 As shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 6 Only one is shown in the figure), a memory 61, in which a computer program 62 that can be run on the processor 60 is stored. When the processor 60 executes the computer program 62, the steps in the above method embodiment are implemented, such as Figure 1 Alternatively, when the processor 60 executes the computer program 62, the functions of the modules / units in the above-mentioned device embodiments are realized.

[0095] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 6. In other embodiments of the present application, the electronic device 6 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0096] The electronic device 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6It is only an example of the electronic device 6 and does not constitute a limitation of the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include an input sending device, a network access device, a bus, etc.

[0097] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0098] The processor 60 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 60 is a cache memory. The memory may store instructions or data that the processor 60 has just used or circulated. If the processor 60 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 60, and thus improves the efficiency of the system.

[0099] In some embodiments, the memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. The memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. Further, the memory 61 may include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program. The memory 61 may also be used to temporarily store data that has been sent or is to be sent.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0101] It should be noted that the structure of the above-mentioned electronic device is only illustrative, and based on different application scenarios, it may also include other physical structures, and the physical structure of the electronic device is not limited here.

[0102] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0103] An embodiment of the present application also provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device implements the steps in the above-mentioned method embodiments.

[0104] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0105] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0106] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0107] The devices, electronic devices, chip systems, computer storage media, and computer program products provided in the above-mentioned embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the methods provided above, and will not be repeated here.

[0108] It should be understood that the above is only to help those skilled in the art to better understand the embodiments of the present application, rather than to limit the scope of the embodiments of the present application. According to the above examples given, those skilled in the art can obviously make various equivalent modifications or changes. For example, some steps in each embodiment of the above detection method may be unnecessary, or some steps may be newly added. Or a combination of any two or any multiple embodiments of the above. Such modifications, changes or combined schemes also fall within the scope of the embodiments of the present application.

[0109] It should also be understood that the division of the methods, situations, categories and embodiments in the embodiments of the present application is only for the convenience of description and should not constitute a special limitation. The features of various methods, categories, situations and embodiments can be combined without contradiction.

[0110] It should also be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0111] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0112] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0115] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for fine-tuning a large language model, characterized in that: The method comprises: In the i-th fine-tuning stage, the parameter space of the large language model is sampled to obtain the model parameter vector; Based on the model parameter vector and the zero-order scale parameter, a differentially private zero-order gradient of a stage loss function is calculated; the stage loss function is obtained by updating the loss function of the large language model; Based on the differential privacy zero-order gradient and the learning rate, iteratively process the model parameters of the large language model according to a preset number of iterations to obtain the target model parameters of the i-th fine-tuning stage; Among them, the target model parameters of the i-th fine-tuning stage are used to construct the parameter space of the i+1-th fine-tuning stage, the iteration round of the i+1-th fine-tuning stage is greater than the iteration round of the i-th fine-tuning stage, the zero-order scale parameter of the i-th fine-tuning stage is less than the zero-order scale parameter of the i+1-th fine-tuning stage, the learning rate of the i-th fine-tuning stage is greater than the learning rate of the i+1-th fine-tuning stage, and i is an integer greater than or equal to 1.

2. The method according to claim 1, characterized in that In the i-th fine-tuning stage, the parameter space of the large language model is sampled to obtain a model parameter vector, including: In the i-th fine-tuning stage, the parameter space of the language model is sampled based on a fixed pruning rate, a dynamic pruning rate, or an incremental pruning rate to obtain the model parameter vector.

3. The method according to claim 1, characterized in that The calculating, based on the model parameter vector and the zero-order scale parameter, the differentially private zero-order gradient of the stage loss function includes: Based on the model parameter vector and the zero-order scale parameter, a differentially private zero-order gradient of the stage loss function is calculated by a zero-order gradient estimation algorithm with a noise term inserted.

4. The method according to claim 3, characterized in that The expression of the zero-order gradient estimation algorithm with inserted noise term is: in, is the differential privacy zero-order gradient function, is the stage loss function, θ is the model parameter, β is the zero-order scale parameter, v is the model parameter vector, and noise is a Gaussian distribution. The noise term.

5. The method according to claim 1, characterized in that Before calculating the differentially private zero-order gradient of the phase loss function, the method further includes: In the i-th fine-tuning stage, updating the loss function of the large language model based on the regularization term to obtain the stage loss function; Among them, the expression of the loss function of updating the large language model based on the regularization term is: is the stage loss function, is the loss function, are the model parameters before fine-tuning at the i-th fine-tuning stage, is the regularization term, and λ is the regularization constant.

6. The method according to claim 1, characterized in that Before calculating the differentially private zero-order gradient of the phase loss function, the method further includes: updating the zero-order scale parameter based on the changes in the fine-tuning phase; Wherein, the expression for updating the zero-order scale parameter is: is the zero-order scale parameter of the i+1th fine-tuning stage; is the zero-order scale parameter of the i-th fine-tuning stage; k is the update coefficient, which is greater than 1.

7. The method according to claim 1, characterized in that Before calculating the differentially private zero-order gradient of the phase loss function, the method further includes: updating the learning rate based on changes during the fine-tuning phase; Among them, the expression for updating the learning rate is: is the learning rate of the i+1th fine-tuning stage; is the learning rate of the i-th fine-tuning stage.

8. The method according to any one of claims 1 to 7, characterized in that: The iterative processing of the model parameters of the large language model according to a preset iteration round based on the differential privacy zero-order gradient and the learning rate to obtain the target model parameters of the i-th fine-tuning stage includes: Calculate the differentially private zero-order gradient of the stage loss function according to a preset number of iterations; Based on the differential privacy zero-order gradient calculated in each iteration round and the learning rate of the i-th fine-tuning stage, the model parameters of the large language model are updated to obtain the target model parameters of the i-th fine-tuning stage.

9. The method according to claim 8, characterized in that Based on the differential privacy zero-order gradient calculated in each iteration round and the learning rate of the i-th fine-tuning stage, the expression for updating the model parameters of the large language model is: in, is the model parameter of the tth iteration of the i-th fine-tuning phase, is the model parameter of the t-1th iteration of the i-th fine-tuning phase, is the learning rate of the i-th fine-tuning stage, is the differentially private zero-order gradient corresponding to the stage loss function of the t-1th iteration round, and t is an integer greater than 1.

10. An electronic device, characterized in that: The electronic device includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method described in any one of claims 1 to 9.

Images