Large model training method, medium and system based on localized differential privacy

Abstract

The present invention discloses a large model training method, medium and system based on localized differential privacy, wherein the method includes: a user terminal device obtains historical data and preprocesses the historical data; a noise distribution function with fixed upper and lower bounds is constructed based on the Lagrange interpolation method and the Fourier series method; the preprocessed historical data is perturbed according to the noise distribution function to generate training data; the user terminal device sends the training data to a server; the server performs model training according to the training data to generate a corresponding large language model. It can effectively improve the quality of data obtained by the localized differential privacy mechanism, and thus improve the availability of the model trained based on the data.

Description

Technical Field

[0001] The present application relates to the field of data privacy protection technology, and in particular to a large model training method, medium and system based on localized differential privacy. Background Art

[0002] Local differential privacy is a privacy protection mechanism that perturbs user data at the source of data collection, ensuring that personal sensitive information is not leaked during data transmission and processing.

[0003] In related technologies, when using local differential privacy mechanisms to train large models, the added noise is often unbounded, causing the perturbed data to deviate from the original value, seriously affecting the availability of the data. Data distortion will reduce the training effect of the model and affect the availability of the final trained model. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, one purpose of the present invention is to propose a large model training method based on localized differential privacy, which can effectively improve the quality of data obtained by the localized differential privacy mechanism, and thus improve the usability of the model trained based on the data.

[0005] In the first aspect, an embodiment of the present invention proposes a large model training method based on localized differential privacy, comprising the following steps: a user terminal device obtains historical data and preprocesses the historical data; a noise distribution function with fixed upper and lower bounds is constructed based on the Lagrange interpolation method and the Fourier series method; the preprocessed historical data is perturbed according to the noise distribution function to generate training data; the user terminal device sends the training data to a server; the server performs model training according to the training data to generate a corresponding large language model.

[0006] According to the large model training method based on localized differential privacy of an embodiment of the present invention, first, the user terminal device obtains historical data and preprocesses the historical data; then, a noise distribution function with fixed upper and lower bounds is constructed based on the Lagrange interpolation method and the Fourier series method; then, the preprocessed historical data is perturbed according to the noise distribution function to generate training data; then, the user terminal device sends the training data to the server; then, the server performs model training according to the training data to generate a corresponding large language model. Thereby, the quality of the data obtained by the local differential privacy mechanism is effectively improved, and the availability of the model trained based on the data is improved.

[0007] In some embodiments, preprocessing the historical data includes data cleaning and data standardization of the historical data; wherein the data cleaning includes missing value processing, outlier detection and data type conversion; the data standardization includes determining the value range of the historical data, mapping the historical data to a preset value interval, and checking the mapped historical data.

[0008] In some embodiments, a noise distribution function with fixed upper and lower bounds is constructed based on Lagrange interpolation method and Fourier series method, including: constructing a basic noise curve based on the Lagrange interpolation method; constructing an enhanced noise curve based on the Fourier series method; adding the basic noise curve and the enhanced noise curve to obtain the noise distribution function.

[0009] In some embodiments, constructing a basic noise curve based on the Lagrangian interpolation method includes: selecting key points and constructing Lagrangian difference polynomials based on the key points; adjusting the parameter values ​​of the key points so that the final basic noise curve meets the probability density function requirements.

[0010] In some embodiments, the Lagrangian difference polynomial is expressed by the following formula:

[0011] ;

[0012] in, represents the Lagrange difference polynomial, x represents the independent variable, represents the ordinate of point i, represents the Lagrangian basis function;

[0013] ;

[0014] in, represents the horizontal coordinate of point i, Represents the horizontal coordinate of point j.

[0015] In some embodiments, the enhanced noise curve is expressed by the following formula:

[0016] ;

[0017] in, represents the enhanced noise curve, represents the half width of the enhanced noise curve, k represents the amplitude of the enhanced noise curve, m represents the frequency of the enhanced noise curve, Represents the coefficient.

[0018] In some embodiments, the expression of the enhanced noise curve is Fourier-series expanded by the following formula:

[0019] ;

[0020] ;

[0021] ;

[0022] in, and represents the coefficient, n represents the nth harmonic of Fourier expansion, x represents the independent variable, nx represents the angle of the trigonometric function at the nth term, and N represents the total number of terms.

[0023] In some embodiments, the amplitude of the enhanced noise curve is calculated by the following formula:

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] in, represents the area of ​​the enhanced noise curve, Represents the area of ​​the basic noise curve.

[0029] In the second aspect, an embodiment of the present invention proposes a computer-readable storage medium, on which a large model training program based on localized differential privacy is stored. When the large model training program based on localized differential privacy is executed by a processor, the large model training method based on localized differential privacy as described above is implemented.

[0030] In the third aspect, an embodiment of the present invention proposes a large model training system based on localized differential privacy, including: a user terminal device, wherein the user terminal device is used to obtain historical data and preprocess the historical data; the user terminal device is also used to construct a noise distribution function with fixed upper and lower bounds based on the Lagrange interpolation method and the Fourier series method, and perturb the preprocessed historical data according to the noise distribution function to generate training data; the user terminal device is also used to send the training data to a server; a server, wherein the server is used to perform model training according to the training data to generate a corresponding large language model.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1is a flowchart of a large model training method based on localized differential privacy according to an embodiment of the present invention;

[0033] Figure 2 is a block diagram of a large model training system based on localized differential privacy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0035] The following describes a large model training method based on localized differential privacy according to an embodiment of the present invention with reference to the accompanying drawings.

[0036] See also Figure 1 , Figure 1 : is a flow chart of a large model training method based on localized differential privacy according to an embodiment of the present invention. Figure 1 As shown, the large model training method based on localized differential privacy includes the following steps:

[0037] S101, the user terminal device obtains historical data and pre-processes the historical data.

[0038] In some embodiments, preprocessing historical data includes data cleaning and data standardization of the historical data; wherein data cleaning includes missing value processing, outlier detection and data type conversion; data standardization includes determining the value range of historical data, mapping the historical data to a preset value interval, and checking the mapped historical data.

[0039] As an example, first, collect the user's raw data from the user's terminal device (for example, the user's smartphone, computer, tablet, etc.) , where N represents the number of users or data items. The data type can be numerical data, text data, behavior records, etc., and there is no restriction on the data type.

[0040] It should be noted that the data collected under the local differential privacy mechanism often involves user privacy. Therefore, the data collection operation should be carried out under the user's authorization and comply with relevant laws and regulations. The original data should be processed locally on the user's terminal device to avoid leakage of the original data.

[0041] Next, the collected historical data is preprocessed; first, missing value processing is performed, that is, for missing data, the missing values ​​are processed by filling the mean, filling the median or deleting the data; then, outlier detection is performed, that is, using statistical methods (such as box plot method, Principle) to detect and handle outliers; then, perform data type conversion, that is, convert the data into a format suitable for subsequent processing; for example, convert the date format into a timestamp, etc.

[0042] Then, data normalization is performed, that is, the data is mapped to a uniform interval to facilitate subsequent data processing and ensure the effectiveness of the noise mechanism.

[0043] Specifically, first, determine the range of the original historical data, that is, set the value range of the original historical data, for example, set the value range to ,in, and are the minimum and maximum values ​​of the data respectively; then, the original historical data Map to Interval, the mapping formula is:

[0044] ;

[0045] in, Represents the mapped data.

[0046] Then, check the mapped data to ensure that all , if there is data out of range, the legitimacy of the mapping formula or the original historical data needs to be checked.

[0047] It should be noted that and The determination of should be based on the understanding of the distribution of historical data to avoid using outliers as upper and lower bounds; and, during the mapping process, attention should be paid to the accuracy of numerical calculations to avoid data errors caused by floating-point operations.

[0048] S102, constructing a noise distribution function with fixed upper and lower bounds based on Lagrange interpolation method and Fourier series method.

[0049] In some embodiments, a noise distribution function with fixed upper and lower bounds is constructed based on Lagrange interpolation and Fourier series method, including: constructing a basic noise curve based on Lagrange interpolation; constructing an enhanced noise curve based on Fourier series method; adding the basic noise curve to the enhanced noise curve to obtain the noise distribution function.

[0050] In some embodiments, constructing a basic noise curve based on the Lagrangian interpolation method includes: selecting key points and constructing Lagrangian difference polynomials based on the key points; adjusting parameter values ​​of the key points so that the final basic noise curve meets the probability density function requirements.

[0051] In some embodiments, the Lagrangian difference polynomial is expressed by the following formula:

[0052] ;

[0053] in, represents the Lagrange difference polynomial, x represents the independent variable, represents the coordinates of point i, represents the Lagrangian basis function;

[0054]

[0055] in, represents the horizontal coordinate of point i, Represents the horizontal coordinate of point j.

[0056] In some embodiments, the enhanced noise curve is expressed by the following formula:

[0057] ;

[0058] in, represents the enhanced noise curve, represents the half width of the enhanced noise curve, k represents the amplitude of the enhanced noise curve, m represents the frequency of the enhanced noise curve, Represents the coefficient.

[0059] In some embodiments, the expression of the enhanced noise curve is Fourier-series expanded by the following formula:

[0060] ;

[0061] ;

[0062] ;

[0063] in, and represents the coefficient, n represents the nth harmonic of Fourier expansion, x represents the independent variable, nx represents the angle of the trigonometric function at the nth term, and N represents the total number of terms.

[0064] In some embodiments, the amplitude of the enhanced noise curve is calculated by the following formula:

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] in, represents the area of ​​the enhanced noise curve, Represents the area of ​​the basic noise curve.

[0070] As an example, first, in order to construct the basic noise curve, five key points are selected , these points will be used to construct the quartic polynomial using the Lagrange interpolation method.

[0071] Specifically, the selection of key points may include: left endpoint, , the probability density is 0; the right endpoint, , the probability density is 0; the point near the left end, , where t is a small positive number and q is a parameter to be determined; the point near the right end, , which is symmetrical about ; Center point, , the probability density is p, p is the parameter to be determined; among them, the value of t is: .

[0072] The Lagrange difference polynomial is expressed by the following formula:

[0073] ;

[0074] in, represents the Lagrange difference polynomial, x represents the independent variable represents the coordinates of point i, represents the Lagrangian basis function;

[0075] ;

[0076] in, represents the horizontal coordinate of point i, Represents the horizontal coordinate of point j.

[0077] The specific calculation process is as follows: First, calculate each basis function ; Then, the corresponding Substituting in and summing up, we get the Lagrange difference polynomial.

[0078] Then, in order to make the basic noise curve meet the requirements of the probability density function, the parameters p and q need to be determined.

[0079] Specifically, the non-negativity condition is set to: for The normalization condition is set to make the area of ​​the basic noise curve meet the normalization requirements, which is expressed by the formula:

[0080] ;

[0081] Thus, by adjusting the values ​​of parameters p and q, the above conditions are satisfied at the same time. Numerical methods or trial and error methods can be used to determine the appropriate parameter values.

[0082] Next, an enhanced noise curve is constructed based on the Fourier series method to increase the probability density near the input data and improve the availability of the disturbed historical data.

[0083] The enhanced noise curve is described by the following formula:

[0084] ;

[0085] in, represents the enhanced noise curve, represents the half width of the enhanced noise curve, represents the amplitude of the enhanced noise curve, represents the frequency of the enhanced noise curve, Represents the coefficient.

[0086] because is a piecewise function, so its expression can be expanded through Fourier series.

[0087] First, calculate coefficient:

[0088] ;

[0089] because exist When it is zero, the integral interval can be reduced to .

[0090] Next, calculate coefficient:

[0091] ;

[0092] Similarly, the integration interval is .

[0093] In the specific calculation process, we can Substitute the expression into the equation and calculate the integral; since The sin function is included, so the product formula of trigonometric functions can be used to simplify the integral calculation; and the calculation result needs to be calculated for each To solve the problem, you can write a program to implement batch operations.

[0094] Then, perform Fourier series expansion:

[0095]

[0096] In practical applications, a finite term N is used for approximation:

[0097] ;

[0098] In order to meet the normalization requirement of the probability density function, the amplitude parameter of the enhancement curve needs to be determined.

[0099] First, calculate the area of ​​the enhancement curve:

[0100] ;

[0101] Next, calculate the integral:

[0102] ;

[0103] The normalization condition is:

[0104] ;

[0105] Then, solve for the amplitude parameter:

[0106] ;

[0107] verify ;make sure , and in superior .

[0108] Next, add the base noise curve and the enhanced noise curve to get the final noise distribution function:

[0109] ;

[0110] Verification conditions are: Non-negativity conditions: for ;

[0111] The normalization condition is: .

[0112] It should be noted that in the numerical calculation process, when calculating the integral and Fourier coefficients, complex numerical calculations may be involved, so it is preferred to use high-precision numerical integration methods or mathematical software (for example, MATLAB, Python's SciPy library) for calculation. In addition, if the above verification conditions cannot be met, the parameters p, q, k and m can be readjusted and the above steps can be repeated for adjustment.

[0113] In some embodiments, the noise mechanism is also verified to satisfy local differential privacy. That is, it is proved that for any adjacent input , and any output , both have:

[0114] ;

[0115] in, It represents the privacy budget, that is, the strength of privacy protection.

[0116] Specifically, first, analyze and relationship, due to The non-zero region of ,when and When very close, The range of variation of is limited; therefore, first calculate the probability density ratio:

[0117] ;

[0118] Taking the worst case, assuming reaches the maximum value k, and is zero.

[0119] Derive the inequality:

[0120] ;

[0121] To ensure that the inequality holds, we adjust k and The minimum value of , so that:

[0122] ;

[0123] Parameter constraints:

[0124] ;

[0125] in, express The minimum value of .

[0126] Thus, by choosing appropriate parameters, the noise mechanism can be satisfy - Localized differential privacy.

[0127] It should be noted that in practical applications, appropriate parameters should be selected according to the required privacy protection strength. In order to ensure the rigor of the proof, a more detailed mathematical derivation can be performed, or numerical simulation can be used to verify the validity of the inequality.

[0128] S103, perturbing the preprocessed historical data according to the noise distribution function to generate training data.

[0129] As an example, when the noise distribution function is obtained Afterwards, the mapped data Perform disturbance to obtain disturbed data .

[0130] Specifically, first, generate the cumulative distribution function:

[0131] ;

[0132] In order to facilitate calculation, we can Discretized and stored as a lookup table.

[0133] Next, generate uniform random numbers, that is, for each , generates a uniform random number in the interval [0,1] .

[0134] Then, perform inverse transform sampling, that is, use the inverse transform method to solve of .

[0135] because It may not be possible to obtain an analytical expression, so numerical methods can be used to solve , such as the bisection method and the Newton-Raphson method.

[0136] Finally, the data is inverse mapped, that is, the perturbed data Mapping back to the original data interval :

[0137]

[0138] It should be noted that in the process of numerical solution, appropriate accuracy requirements need to be set to ensure To ensure the accuracy of the prediction, for large-scale data, parallel computing or optimization algorithms can be used to improve the computing efficiency; and statistical analysis can be performed on the disturbed data to verify whether the distribution of the detector meets expectations.

[0139] S104, the user terminal device sends the training data to the server.

[0140] In some embodiments, the training data (i.e., the perturbed data) is transmitted to a server or a model training center via a secure communication protocol.

[0141] S105: The server performs model training according to the training data to generate a corresponding large language model.

[0142] In some embodiments, the training of the model includes training data preparation, model construction and model training, data set division, that is, dividing the perturbed data set into training set, validation set and test set; data format conversion, that is, converting the data format according to the model requirements, for example, word segmentation and encoding of text data; then, batch processing, that is, dividing the data into small batches for batch processing during model training; then, model construction, first select the model architecture, and select the appropriate large language model architecture according to the task requirements, such as Transformer, GPT-3, etc. Then, set the model parameters: including the number of layers, the number of hidden units, the number of attention heads, the activation function, etc. Then, initialize the model parameters: use a suitable parameter initialization method, such as Xavier initialization and He initialization. Then, the training process of the specific model includes: first, setting the training hyperparameters: preferably, the learning rate α, for example, α=0.0001; the batch size B, for example, B=32; the number of training rounds E, for example, E=20; the optimization algorithm, such as Adam, RMSProp, etc.; then, the training loop, training E rounds: for each batch of data, the training includes the following steps: first, forward propagation is performed to calculate the model output; then the loss is calculated using an appropriate loss function (such as cross entropy loss); then the gradient of the loss with respect to the model parameters is calculated; then the model parameters are updated according to the optimization algorithm; then, to prevent the gradient from exploding, the gradient is clipped; then, the indicators are recorded, that is, the training loss, accuracy and other indicators are recorded; in this way, the training of a batch of data is completed; then, at a certain number of steps, the model performance is evaluated on the validation set, the hyperparameters are adjusted, and based on the validation set performance, it is decided whether to stop the training early to prevent overfitting.

[0143] It should be noted that hyperparameter tuning: adjust hyperparameters such as learning rate and batch size according to training results to obtain the best model performance. Training log: record information such as loss, accuracy, and learning rate changes during training for easy analysis. Model saving: save the model checkpoint during training for subsequent loading and reasoning.

[0144] In some embodiments, after the model training is completed, the performance of the model is also evaluated:

[0145] 1. Test set evaluation: Evaluate model performance on the test set and calculate indicators such as accuracy, precision, recall, and F1 value.

[0146] 2. Task-related indicator evaluation: According to the specific task, calculate the corresponding indicators, such as the perplexity of the language model and the BLEU score of machine translation.

[0147] 3. Error analysis: Analyze the error cases of the model and find possible improvements.

[0148] In some embodiments, after the model training is completed, the privacy protection effect of the model is also evaluated:

[0149] 1. Member reasoning attack test:

[0150] (a) Construct a membership inference attack model to try to determine whether a specific data is used for model training. (b) Evaluate the model's ability to resist membership inference attacks and verify the effectiveness of localized differential privacy. (c) Calculate the attack success rate. Ideally, the attack success rate should be close to the level of random guessing.

[0151] 2. Attribute reasoning attack test:

[0152] (a) Evaluate whether the model leaks the attribute information of the training data. (b) Attack the model and try to infer certain attributes of the training data. (c) Evaluate the attack success rate and verify the privacy protection effect of the model.

[0153] 3. Differential privacy verification:

[0154] (a) Theoretical analysis of the privacy protection capability of the noise mechanism. (b) Experimental proof that the output of the verification model does not depend on the presence or absence of a single data point.

[0155] In summary, according to the large model training method based on localized differential privacy in an embodiment of the present invention, first, the user terminal device obtains historical data and preprocesses the historical data; then, a noise distribution function with fixed upper and lower bounds is constructed based on the Lagrange interpolation method and the Fourier series method; then, the preprocessed historical data is perturbed according to the noise distribution function to generate training data; then, the user terminal device sends the training data to the server; then, the server performs model training based on the training data to generate a corresponding large language model. Thereby, the quality of the data obtained by the local differential privacy mechanism is effectively improved, and the availability of the model trained based on the data is improved.

[0156] In the second aspect, an embodiment of the present invention proposes a computer-readable storage medium, on which a large model training program based on localized differential privacy is stored. When the large model training program based on localized differential privacy is executed by a processor, the large model training method based on localized differential privacy as described above is implemented.

[0157] Thirdly, Figure 2 As shown, an embodiment of the present invention proposes a large model training system based on localized differential privacy, including: a user terminal device 10 and a server 20.

[0158] The user terminal device 10 is used to obtain historical data and pre-process the historical data;

[0159] The user terminal device 10 is also used to construct a noise distribution function with fixed upper and lower bounds based on the Lagrange interpolation method and the Fourier series method, and to perturb the preprocessed historical data according to the noise distribution function to generate training data;

[0160] The user terminal device 10 is also used to send the training data to the server 20;

[0161] The server 20 is used to perform model training according to the training data to generate a corresponding large language model.

[0162] It should be noted that the above description of the large model training method based on localized differential privacy is also applicable to the large model training system based on localized differential privacy, and will not be repeated here.

[0163] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0164] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0165] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0166] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0167] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0168] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0169] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0170] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A large model training method based on localized differential privacy, characterized in that: The following steps are involved: The user terminal device obtains historical data and pre-processes the historical data; Construct a noise distribution function with fixed upper and lower bounds based on Lagrange interpolation and Fourier series methods; Perturbing the preprocessed historical data according to the noise distribution function to generate training data; The user terminal device sends the training data to a server; The server performs model training according to the training data to generate a corresponding large language model; Among them, the noise distribution function with fixed upper and lower bounds is constructed based on the Lagrange interpolation method and the Fourier series method, including: Constructing a basic noise curve based on the Lagrange interpolation method; Constructing an enhanced noise curve based on the Fourier series method; Adding the basic noise curve and the enhanced noise curve to obtain the noise distribution function; The enhanced noise curve is expressed by the following formula: in, represents the enhanced noise curve, represents the half width of the enhanced noise curve, represents the amplitude of the enhanced noise curve, represents the frequency of the enhanced noise curve, represents the coefficient; The amplitude of the enhanced noise curve is calculated by the following formula: in, represents the area of ​​the enhanced noise curve, Represents the area of ​​the basic noise curve.

2. The large model training method based on localized differential privacy as claimed in claim 1, characterized in that: Preprocessing the historical data includes data cleaning and data standardization of the historical data; Wherein, the data cleaning includes missing value processing, outlier detection and data type conversion; The data standardization includes determining a value range of historical data, mapping the historical data to a preset value interval, and checking the mapped historical data.

3. The large model training method based on localized differential privacy as claimed in claim 1, characterized in that: The basic noise curve is constructed based on the Lagrange interpolation method, including: Selecting key points, and constructing Lagrange difference polynomials based on the key points; The parameter values ​​of the key points are adjusted so that the final basic noise curve meets the requirements of the probability density function.

4. The large model training method based on localized differential privacy as claimed in claim 3, characterized in that: The Lagrange difference polynomial is expressed by the following formula: in, represents the Lagrange difference polynomial, represents the independent variable, express The vertical coordinate of the point, represents the Lagrangian basis function; in, represents the horizontal coordinate of point i, Represents the horizontal coordinate of point j.

5. The large model training method based on localized differential privacy as claimed in claim 1, characterized in that: The expression of the enhanced noise curve is expanded in Fourier series by the following formula: in, and represents the coefficient, represents the Fourier expansion of harmonics, represents the independent variable, Represents the trigonometric function in The angle of the item, Indicates the total number of items.

6. A computer-readable storage medium, characterized in that: A large model training program based on localized differential privacy is stored thereon, and when the large model training program based on localized differential privacy is executed by a processor, the large model training method based on localized differential privacy as described in any one of claims 1-5 is implemented.

7. A large model training system based on localized differential privacy, characterized in that: include: A user terminal device, the user terminal device is used to obtain historical data and pre-process the historical data; The user terminal device is also used to construct a noise distribution function with fixed upper and lower bounds based on Lagrange interpolation method and Fourier series method, and to perturb the preprocessed historical data according to the noise distribution function to generate training data; The user terminal device is also used to send the training data to the server; A server, wherein the server is used to perform model training according to the training data to generate a corresponding large language model; Among them, the noise distribution function with fixed upper and lower bounds is constructed based on the Lagrange interpolation method and the Fourier series method, including: Constructing a basic noise curve based on the Lagrange interpolation method; Constructing an enhanced noise curve based on the Fourier series method; Adding the basic noise curve and the enhanced noise curve to obtain the noise distribution function; The enhanced noise curve is expressed by the following formula: in, represents the enhanced noise curve, represents the half width of the enhanced noise curve, represents the amplitude of the enhanced noise curve, represents the frequency of the enhanced noise curve, represents the coefficient; The amplitude of the enhanced noise curve is calculated by the following formula: in, represents the area of ​​the enhanced noise curve, Represents the area of ​​the basic noise curve.

Images