DPO-Tune method for fine tuning and reasoning stages of pre-trained large language model

Abstract

The invention relates to a DPO-Tune method for a fine tuning and reasoning stage of a pre-trained large language model, and the method comprises the following steps: S1, adding a Renyi divergence optimizer in a cGAN, and carrying out the training, and forming an R-cGAN; s2, adding a dynamic disturbance factor function and an R-cGAN in the random response mechanism to generate a dual random response mechanism; s3, processing the sequence information in the to-be-disturbed data set by using DP-Forward, and adding disturbance to the sequence information in the to-be-disturbed data set according to an analytic matrix Gaussian mechanism in a data forward transmission process; and disturbing the feature tag data pair in the to-be-disturbed data set by using a dual random response mechanism. According to the method, the privacy of the sequence and the label information can be effectively improved on the premise of ensuring the accuracy of the large language model, so that the availability of the model is increased, and the balance between the accuracy and the privacy of the model is realized.

Description

Technical Field

[0001] The present invention relates to a network privacy protection method, in particular to a DPO-Tune method for fine-tuning and reasoning stages of a pre-trained large language model. Background Art

[0002] Some existing studies on DP-NLP directly use differential privacy technology, which leads to a serious loss of model accuracy because the model parameters of large language models are large. Therefore, the main goal of existing research is to implement DP in natural language processing tasks to improve user privacy in large language models while improving data availability and the accuracy of analysis results to ensure a trade-off between privacy and utility. The existing mainstream research methods mainly add randomness through gradient perturbation methods (such as DP-SGD) to ensure differential privacy. However, the addition of noise will lead to a decrease in model performance, especially when processing high-dimensional data or complex models, the defects of DP-SGD are particularly obvious.

[0003] In addition, LDP, as an improvement of DP, is used to protect data during the data collection phase and is also used to fine-tune large language models. In this vein, the DP-Forward method was proposed, which perturbs the embedding during the forward pass of the large language model. DP-Forward is based on Sequence Local Differential Privacy (Seq-LDP), which locally adds noise to the embedding of the sequence and implements it through the analytic matrix Gaussian mechanism. Therefore, DP-Forward provides qualified protection for sequence information and guarantees the accuracy of the model. However, the defect of the DP-Forward scheme is that it does not provide sufficient protection for label information. Specifically, the DP-Forward method believes that sequence information contains richer privacy information, while label information is relatively easier to protect, so a random response mechanism is used to simply perturb the label. However, the current methods mostly remain at the traditional stage of understanding labels. For example, the DP-Forward method assumes that label information usually depends on their sequence, so even if the label information is protected by any label-DP mechanism, it is still possible to recover the true label from the original sequence, which is why researchers believe that it is meaningful to protect only sequence information. At the same time, subsequent schemes also lack effective protection for label information.

[0004] In summary, the protection of label information in existing schemes is based on traditional label differential privacy. Traditional label differential privacy is a relaxation of differential privacy. It protects each trained label, but cannot protect users from label inference attacks. The traditional definition of label differential privacy does not consider the correlation between labels and features, and generally assumes that labels are public or sensitive. When only label information is protected, attackers may obtain label information by attacking features. Protecting the privacy of both labels and features is redundant for some features, which will cause the accuracy of the model to drop significantly and the model utility cannot be guaranteed. Based on this conclusion, existing schemes represented by DP-Forward have loopholes in protecting label information based on traditional label differential privacy. Most existing schemes cannot resist label inference attacks, resulting in user privacy leakage. In addition, although the existing schemes that consider strict label privacy protection use a double random response mechanism to generate synthetic features related to labels, the effect of synthetic features is not accurate and does not achieve a trade-off between privacy and utility. Summary of the invention

[0005] The purpose of the present invention is to provide a DPO-Tune method for fine-tuning and inference stages of a pre-trained large language model to solve the problem that the method of simultaneously protecting the differential privacy level of labels and sequences during the forward pass cannot achieve a trade-off between privacy and utility.

[0006] The present invention is achieved in that:

[0007] A DPO-Tune method for fine-tuning and inference phases of a pre-trained large language model, comprising the following steps:

[0008] S1. Add the Rényi divergence optimizer to cGAN and train it to form R-cGAN;

[0009] S2. Add the dynamic perturbation factor function and R-cGAN to the random response mechanism to generate a double random response mechanism;

[0010] S3. Use DP-Forward to process the sequence information in the dataset to be disturbed, and add disturbance to the sequence information in the dataset to be disturbed according to the analytical matrix Gaussian mechanism during the forward transmission of data; use the double random response mechanism to perturb the feature label data pairs in the dataset to be disturbed.

[0011] Furthermore, the specific method of forming R-cGAN in step S2 is:

[0012] S1-1. Add a variational Rényi divergence optimizer to cGAN and pre-set the parameters of the variational Rényi divergence optimizer.

[0013] S1-2. Select original feature label data pairs in the training set, randomly respond to the original labels in the original feature label data pairs, generate perturbation labels, input the perturbation labels into cGAN, and obtain synthetic features;

[0014] S1-3. Calculate the difference between the synthesized features and the original features in the original feature label data pair according to the Rényi divergence optimizer of the variational formula;

[0015] S1-4. If the difference is greater than the preset value, change the parameters of the Rényi divergence optimizer of the variational formula, repeat steps S1-2 and S1-3 to randomly respond to the original labels in the original feature label data pair, and generate perturbation labels; input the perturbation labels into cGAN to obtain synthetic features, calculate the difference between the synthetic features and the original features in the original feature label data pair, until the difference reaches the preset value, and use the corresponding parameters when the difference reaches the preset value as the parameters of the Rényi divergence optimizer of the variational formula to obtain R-cGAN.

[0016] Furthermore, the calculation formula of the Rényi divergence optimizer based on the variational formula is:

[0017]

[0018] in, and is the normalization factor in the definition of Rényi divergence, Δk is a k-dimensional real vector, T is a sufficient statistic used to describe data characteristics, Q is the actual data distribution, P is the distribution of the data after cGAN, dQ is the measure relative to distribution Q, and dP is the measure relative to distribution P.

[0019] Furthermore, step S2 includes the following sub-steps:

[0020] S2-1. Replace the flipping probability in the random response mechanism with a dynamic perturbation factor function;

[0021] S2-2. After the random response, R-cGAN is added, and the perturbed labels are input into R-cGAN. According to the original features in the original feature-label data pair, the synthesized features are generated to obtain the perturbed feature-label data pair.

[0022] Furthermore, the calculation formula of the dynamic disturbance factor function is:

[0023] λ(x)=λ0+α·S(x)

[0024] Among them, λ(x) is the dynamic disturbance factor function, λ0 is the basic disturbance factor, α is the sensitivity adjustment coefficient, S(x) is the sensitivity function, and the calculation formula of S(x) is:

[0025]

[0026] Furthermore, the method for determining the basic disturbance factor λ0 and the sensitivity adjustment coefficient α is:

[0027] Input the original label y of the training set and use the double random response mechanism to generate the perturbed feature label data pair The perturbed feature label data is input into the model to simulate the label inference attack. λ0 and α are continuously adjusted according to the attack accuracy and model accuracy until the preset model accuracy is reached and the attack accuracy is less than the preset attack accuracy.

[0028] Furthermore, the specific method of adding disturbance to the sequence information in the data set in step S3 is:

[0029] The calculation formula of the analytical matrix Gaussian mechanism is:

[0030] Pr[∑M,X,X ′ ≥∈]-e ∈ Pr[∑M,X ′ ,X≤-∈]≤δ

[0031] Among them, M is the privacy protection mechanism, X is the sequence information, and X ′ is the sequence information after disturbance, ∈ is the privacy parameter, which is used to measure the degree of privacy loss, and δ is the probability that the mechanism does not satisfy differential privacy.

[0032] Based on the DP-Forward method, the present invention adds disturbance to the sequence information during the forward transmission process to achieve privacy protection of the sequence information. The present invention adds a Rényi divergence optimizer to cGAN, continuously iterates and optimizes, and by discriminating the Rényi difference, enables R-cGAN to generate synthetic features by perturbing the labels, thereby ensuring that the privacy protection effect of the label information is improved without affecting the model performance. The present invention uses a double random response mechanism to require the addition of disturbances to the feature information but does not lose the correlation between the feature and the label information to obtain the synthetic feature. The privacy protection effect of the double random response mechanism depends on the quality of the synthetic feature. R-cGAN improves the quality of the synthetic feature to ensure that attackers cannot use the correlation between the label and the feature to illegally obtain the label information.

[0033] The present invention makes the flip probability λ in the random response mechanism dynamic, and uses the dynamic disturbance factor to optimize the dual random response mechanism. The dynamic disturbance factor can dynamically adjust the disturbance level according to different data properties. For data with low sensitivity requirements, the corresponding overhead can be reduced. For data with high sensitivity requirements, sufficiently strict privacy protection can be provided, so that the present invention can set different sensitive functions according to different data sets and downstream tasks. The present invention can adapt to different data sets, thereby improving the practicality of the present invention. Through experiments, it is found that when the flip probability λ of the present invention is constant, the accuracy rate is still maintained at about 90%. The present invention also obtains a high accuracy rate of the model while ensuring a high level of privacy protection effect, achieving a balance between privacy and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a process framework diagram of the present invention.

[0035] Figure 2 It is the joint disturbance flow chart of the present invention.

[0036] Figure 3 (a) is a schematic diagram comparing the training attack rates of the method of the present invention and other classic privacy protection methods under label inference attacks, and (b) is a schematic diagram comparing the test attack rates of the method of the present invention and other classic privacy protection methods under label inference attacks.

[0037] Figure 4 This is a schematic diagram comparing the model accuracy of the present invention and other classic privacy protection methods.

[0038] Figure 5 It is a comparison chart of the overhead of the dual random response mechanism of the present invention and the other three methods.

[0039] Figure 6 (a) is a comparison chart of the label inference attack rate of the double random response mechanism of the present invention and the other two methods under different flipping probabilities λ, and (b) is a comparison chart of the model accuracy of the double random response mechanism of the present invention and the other two methods under different flipping probabilities λ.

[0040] Figure 7 It is a privacy budget comparison diagram of the method of the present invention under different dynamic flipping probabilities λ(x). DETAILED DESCRIPTION

[0041] The present invention is further described below.

[0042] like Figure 1 As shown, the DPO-Tune method of the present invention for fine-tuning and reasoning of a pre-trained large language model includes the following steps:

[0043] S1. Add the Rényi divergence optimizer to cGAN and train it to form R-cGAN.

[0044] S1-1. To address the label privacy issue, a Rényi divergence optimizer for the variational formula is added to cGAN, and the parameters of the Rényi divergence optimizer for the variational formula are pre-set.

[0045] The Rényi divergence variation formula is:

[0046]

[0047] Among them, R α (Q||P) is the Rényi divergence between distribution Q and distribution P, sup means taking the supremum, Δk is a k-dimensional real vector, T is a sufficient statistic used to describe the data characteristics, Q is the real data distribution, and P is the distribution of the data after passing through cGAN. and is the normalization factor in the definition of the Rényi divergence.

[0048] After obtaining the Rényi divergence variation formula, we first need to calculate R α (Q||P) vs. Δ K The gradient of , and then use the gradient descent algorithm to update the parameter Δ K ,for: Where η is the learning rate of the gradient descent algorithm. The parameters (i.e., the parameters in formula (2)) are continuously updated iteratively until the convergence condition is reached, i.e., the difference reaches the preset value. After finding the optimal Δk, the optimal synthetic distribution can be calculated: Among them, Δβ is a shift parameter to ensure the normalization of the distribution. This helps to output the optimal synthetic features; and then the calculation formula of the Rényi divergence optimizer of the variational formula is obtained as follows:

[0049]

[0050] Here, Δk·T represents how the Δk transformation affects the data T, dQ is the measure relative to the distribution Q, and dP is the measure relative to the distribution P.

[0051] Therefore, the optimization problem of the Rényi divergence variational formula is transformed into finding Δk that maximizes f(Δk), which specifically includes steps S1-2 to S1-4.

[0052] like Figure 2 As shown, S1-2. Divide the data set into a training set and a test set, select a part of the allowed original feature label data pairs (x, y) in the training set, randomly respond to the original label y in the original feature label data pair, and generate a perturbation label Perturb the label And noise z are input into the generator of cGAN to obtain synthetic features

[0053] S1-3. Calculate synthetic features based on the Rényi divergence optimizer of the variational formula The difference between the original feature x and the original feature label data pair.

[0054] The difference between the synthetic feature and the original feature in the original feature label data pair is f(Δk) in formula (2), x and Entering the discriminator, the Rényi divergence is used to determine the difference between them.

[0055] S1-4. If the difference is greater than the preset value, change the parameters of the Rényi divergence optimizer of the variational formula, repeat steps S1-2 and S1-3 to randomly respond to the original labels in the original feature label data pair, and generate perturbation labels; input the perturbation labels into cGAN to obtain synthetic features, calculate the difference between the synthetic features and the original features in the original feature label data pair, until the difference reaches the preset value, and use the corresponding parameters when the difference reaches the preset value as the parameters of the Rényi divergence optimizer of the variational formula to obtain R-cGAN.

[0056] In the present invention, when the difference value is less than or equal to 0.05, the convergence condition is reached.

[0057] The original feature label data randomly responds to the original label to generate a perturbation label. Return the generator to iterate and optimize using the Rényi divergence optimizer until synthetic features with small enough differences can be generated.

[0058] This paper proposes to use the Rényi divergence optimizer to improve cGAN, continuously iterate the optimization, and make cGAN pass the perturbation label by discriminating the Rényi difference. and noise z to generate the optimal synthetic features It ensures the best privacy protection effect without affecting the model performance, that is, it can meet the trade-off between model privacy and efficiency.

[0059] S2. Add the dynamic perturbation factor function and the trained R-cGAN to the random response mechanism to generate a double random response mechanism.

[0060] Since the core idea of ​​the present invention is to use a double random response mechanism to add disturbances to the tag information, it ensures that attackers cannot use the correlation between the tag and the feature to illegally obtain the tag information. Specifically, the double random response mechanism requires adding disturbances to the feature information but does not require losing the correlation between the feature and the tag information, which is called a synthetic feature. Therefore, the privacy protection effect of the double random response mechanism depends on the quality of the synthetic feature.

[0061] The flipping probability in the random response mechanism is replaced by a dynamic perturbation factor function. After the random response, R-cGAN is added, and the perturbed label is input into R-cGAN. According to the original features in the original feature label data pair, the synthesized features are generated to obtain the perturbed feature label data pair, thereby generating a double random response mechanism.

[0062] In order to improve the practicality of the present invention, a suitable sensitivity function S(x) is selected according to the characteristics of different data sets, a dynamic disturbance factor function λ(x) is applied according to S(x), and λ(x) is used in a double random response mechanism.

[0063] Taking a data set used in the present invention as an example, first of all, the key to selecting the sensitivity level of data as an independent variable is to quantify the sensitivity level of the data. To this end, S(x) is set to quantify the sensitivity level of the data. Taking personal identity information and general classified information as an example, for a data set containing these two types of data, importance∈[0,1] is set to adjust the sensitivity. For personal identity information with high sensitivity level requirements, importance is used and the maximum value is limited to 1. For general information, importance*0.5 is used to reduce its sensitivity level. Define S(x) as:

[0064]

[0065] S(x) can be further adjusted according to actual needs to accommodate quantization of different data types and sensitivity levels.

[0066] After quantifying the sensitivity level of the data, a linear function is selected to make the disturbance factor dynamic. For the dynamic disturbance factor function λ(x), the calculation formula of λ(x) is:

[0067] λ(x)=λ0+α·S(x) (4)

[0068] Among them, λ0 is the basic disturbance factor and α is the sensitivity adjustment coefficient.

[0069] According to the expression, a higher λ0 can provide stronger privacy protection. Therefore, for sensitive data, a higher λ0 can be selected, while for relatively insensitive data, a lower λ0 can be selected. When selecting α, the range of variation of the sensitivity function S(x) needs to be considered. If the range of variation is large, a smaller α should be selected to avoid excessive disturbance. Obviously, a larger α can provide stronger privacy protection, but it may reduce the practicality of the data, thereby affecting the performance of the model.

[0070] After adding the dynamic factor function, we add R-cGAN and simulate the label inference attack to find the optimal λ0 and α. We input the original label y of the training set and first generate the corresponding perturbation label according to the random response mechanism. Then use R-cGAN to perturb the label Generate corresponding synthetic features Thus, we can obtain the perturbed feature label data pair The perturbed feature label data is input into the model to simulate label inference attacks. λ0 and α are continuously adjusted according to the attack accuracy and model accuracy until the optimal model privacy and utility are achieved, that is, until the model accuracy reaches the preset model accuracy and the attack accuracy is less than the preset attack accuracy.

[0071] In the basic dual random response mechanism, the perturbation label is input into cGAN to generate a synthetic feature related to the label Synthetic Features The quality of is crucial, so in order to ensure the performance of the model while improving the privacy protection of label information,

[0072] S3. The sequence information in the perturbed data set is processed using DP-Forward, and during the forward transmission of data, perturbations are added to the sequence information in the perturbed data set according to the analytic matrix Gaussian mechanism; the feature label data pairs in the perturbed data set are perturbed using the double random response mechanism and R-cGAN.

[0073] By adding disturbances to the sequence information during the forward data transmission process, the framework DP-Forward of the sequence differential privacy level is implemented. For the sequence information, the analytical matrix Gaussian mechanism aMGM is used to protect the training and inference data while meeting the strict local differential privacy (LDP) requirements. This mechanism meets:

[0074] Pr[∑M,X,X ′ ≥∈]-e ∈ Pr[∑M,X ′ ,X≤-∈]≤δ

[0075] Where M represents a privacy protection mechanism, X is the sequence information, and X ′ is the perturbed sequence information, ∈ is the privacy parameter, which is used to measure the degree of privacy loss and controls how much the distribution of the privacy mechanism output changes under similar input data sets (i.e., two data sets with only one element different); δ is used to indicate the probability that the mechanism does not satisfy differential privacy.

[0076] According to the dynamic factor function, the labels of the feature label data pairs of the data set to be perturbed are perturbed to obtain perturbed labels, and the perturbed labels are input into R-cGAN to generate synthetic features, thereby obtaining the perturbed feature label data pairs in the data set to be perturbed.

[0077] The above adjustment of λ0 and α can be adjusted when using the double random response matrix mechanism to process the perturbed data set, because different data sets have different sensitivities and thus require different values ​​of λ0 and α.

[0078] like Figure 3 As shown in the figure, the horizontal axis represents a variety of privacy protection schemes, specifically the traditional random response mechanism RR, the double random response mechanism DRR, the perturbation of labels and features based on their correlation, the DRR using the Adam optimizer, the DRR using the Adam optimizer with a dynamic perturbation factor, the DRR using the Rényi divergence optimizer, and the method of the present invention. The vertical axis represents the accuracy of label inference attacks in the training phase and the test phase after training fine-tuning is completed. When the flip probability λ is fixed to 0.5, Figure 3 (a) It can be seen that although the risk of label inference attack of the present invention is reduced to about 75% during the training phase, except for the DRR method using the Renyi divergence optimizer, compared with other methods, the method of the present invention has significant advantages and can effectively resist label inference attacks.

[0079] like Figure 3 As shown in (b), after training and fine-tuning, the efficiency of all privacy protection mechanisms in resisting label inference attacks has increased, but the method of the present invention still maintains the best privacy protection effect, which is maintained at about 70%. The DRR method using the Renyi divergence optimizer is close to the effect of the method of the present invention in resisting label inference attacks, but it is not perfectly applicable to multiple types of data sets. The difference is that the present invention uses a dynamic perturbation factor function, and for data with different sensitivity requirements in different data sets, it can dynamically adjust the sensitivity level according to the sensitivity function, saving unnecessary overhead. At the same time, due to the introduction of additional randomness, the present invention is still improved in resisting label inference attacks.

[0080] like Figure 4As shown in the figure, compared with other privacy protection mechanisms, the accuracy of the method of the present invention is still maintained at about 90% under the condition of ensuring a certain flip probability λ, which is very competitive compared with other methods. Figure 4 It can be seen that the idea of ​​perturbing labels and features separately has a significant disadvantage in terms of model accuracy. The reason is that this idea does not consider the correlation between labels and features, but simply perturbs them separately, causing a large deviation in information and resulting in poor fine-tuning training results.

[0081] like Figure 5 As shown, Figure 5 The training time and memory cost comparisons for each batch are shown. The method of perturbing labels and features separately has high requirements for time and memory. Although the RR mechanism ensures low time and memory costs, it follows the traditional definition of privacy in terms of privacy protection, which has been experimentally proven to have obvious defects in privacy protection. In summary, the present invention fully considers the characteristics of label privacy protection and ensures excellent competitiveness in terms of cost.

[0082] After comparing the privacy and accuracy of the method of the present invention with multiple privacy protection mechanisms under the same flipping probability, the present invention aims to explore the privacy and accuracy characteristics of the method under different flipping probabilities.

[0083] like Figure 6 As shown in the figure, the experimental results show that as the flip probability λ decreases, the accuracy of the model decreases, but the model's ability to resist label inference attacks is also significantly improved. The specific reason is that for the original feature label data pair (x, y), for the sample label y, there is:

[0084]

[0085] Final output (x,y i ′ ). Among them, y i ′ Represents the perturbed label data. The increased flip probability means that the label has a greater probability of being perturbed. In this case, the increase in λ means the improvement of privacy protection. The comprehensive comparison of privacy protection effect and accuracy shows that the method of the present invention has the best performance.

[0086] like Figure 7As shown in the figure, after the dynamic perturbation factor function is applied, the privacy budget ε changes with different dynamic perturbation effects. The privacy budget ε represents the amount of private information allowed to be leaked by the model. A smaller privacy budget ε means a stronger privacy protection effect, while a larger privacy budget ε means more privacy leakage. As can be seen from the figure, as the dynamic perturbation effect increases, the privacy budget ε gradually decreases, indicating that while the method of the present invention applies the dynamic perturbation factor function to improve the practicality of the method, it ensures that the privacy protection effect still changes with the dynamic flip probability. The method of the present invention truly dynamically achieves the optimal trade-off between privacy protection and model performance.

Claims

1. A DPO-Tune method for fine-tuning and inference phases of a pre-trained large language model, characterized by: The following steps are involved: S1. Add the Rényi divergence optimizer to cGAN and train it to form R-cGAN; S2. Add the dynamic perturbation factor function and R-cGAN to the random response mechanism to generate a double random response mechanism; S3. Use DP-Forward to process the sequence information in the dataset to be disturbed, and add disturbance to the sequence information in the dataset to be disturbed according to the analytical matrix Gaussian mechanism during the forward transmission of data; use the double random response mechanism to perturb the feature label data pairs in the dataset to be disturbed.

2. The DPO-Tune method for fine-tuning and reasoning of a pre-trained large language model according to claim 1, characterized in that: The specific method of forming R-cGAN in step S2 is: S1-1. Add a variational Rényi divergence optimizer to cGAN and pre-set the parameters of the variational Rényi divergence optimizer. S1-2. Select original feature label data pairs in the training set, randomly respond to the original labels in the original feature label data pairs, generate perturbation labels, input the perturbation labels into cGAN, and obtain synthetic features; S1-3. Calculate the difference between the synthesized features and the original features in the original feature label data pair according to the Rényi divergence optimizer of the variational formula; S1-4. If the difference is greater than the preset value, change the parameters of the Rényi divergence optimizer of the variational formula, repeat steps S1-2 and S1-3 to randomly respond to the original labels in the original feature label data pair, and generate perturbation labels; input the perturbation labels into cGAN to obtain synthetic features, calculate the difference between the synthetic features and the original features in the original feature label data pair, until the difference reaches the preset value, and use the corresponding parameters when the difference reaches the preset value as the parameters of the Rényi divergence optimizer of the variational formula to obtain R-cGAN.

3. The DPO-Tune method for fine-tuning and reasoning of a pre-trained large language model according to claim 2, characterized in that: The calculation formula of the Rényi divergence optimizer based on the variational formula is: in, and is the normalization factor in the definition of Rényi divergence, Δk is a k-dimensional real vector, T is a sufficient statistic used to describe data characteristics, Q is the actual data distribution, P is the distribution of the data after cGAN, dQ is the measure relative to distribution Q, and dP is the measure relative to distribution P.

4. The DPO-Tune method for fine-tuning and reasoning of a pre-trained large language model according to claim 1, characterized in that: Step S2 includes the following sub-steps: S2-1. Replace the flipping probability in the random response mechanism with a dynamic perturbation factor function; S2-2. After the random response, R-cGAN is added, and the perturbed labels are input into R-cGAN. According to the original features in the original feature-label data pair, the synthesized features are generated to obtain the perturbed feature-label data pair.

5. The DPO-Tune method for fine-tuning and reasoning of a pre-trained large language model according to claim 4, characterized in that: The calculation formula of the dynamic disturbance factor function is: λ(x)=λ0+α·S(x) Among them, λ(x) is the dynamic disturbance factor function, λ0 is the basic disturbance factor, α is the sensitivity adjustment coefficient, S(x) is the sensitivity function, and the calculation formula of S(x) is:

6. The DPO-Tune method for fine-tuning and reasoning of a pre-trained large language model according to claim 5, characterized in that: The method for determining the basic disturbance factor λ0 and the sensitivity adjustment coefficient α is: Input the original label y of the training set and use the double random response mechanism to generate the perturbed feature label data pair The perturbed feature label data is input into the model to simulate the label inference attack, and λ0 and α are continuously adjusted according to the attack accuracy and model accuracy until the model accuracy reaches the optimal model accuracy and the attack accuracy is less than the optimal attack accuracy.

7. The DPO-Tune method for fine-tuning and reasoning of a pre-trained large language model according to claim 1, characterized in that: The specific method of adding disturbance to the sequence information in the data set in step S3 is: The calculation formula of the analytical matrix Gaussian mechanism is: Pr[∑M,X,X ′ ≥∈]-e ∈ Pr[∑M,X ′ ,X≤-∈]≤δ Among them, M is the privacy protection mechanism, X is the sequence information, and X ′ is the sequence information after disturbance, ∈ is the privacy parameter, which is used to measure the degree of privacy loss, and δ is the probability that the mechanism does not satisfy differential privacy.

Images