Privacy model training method based on low-rank projection, medium and system

Abstract

The application discloses a privacy model training method based on low-rank projection, a medium and a system, wherein the method comprises the following steps: sampling according to a preset sampling rate to obtain a corresponding sampling batch, and calculating a gradient set; obtaining a periodic index parameter, and judging whether the current period has been completed according to the periodic index parameter; if the current period has been completed, estimating a covariance matrix, extracting a principal direction to obtain an orthogonal basis, calculating a corresponding energy preservation ratio according to the orthogonal basis, and determining a principal subspace according to the energy preservation ratio; calculating a post-projection threshold value according to the energy preservation ratio; calculating an average gradient, and projecting to obtain a corresponding projection gradient; generating a full-dimensional noise, combining the full-dimensional noise to generate a combined noise gradient, and updating a model parameter according to the combined noise gradient; the method can effectively improve the convergence speed in the privacy model training process, improve the model training efficiency, and improve the accuracy of the final model prediction result.

Description

Technical Field

[0001] This application relates to the field of privacy model technology, and in particular to a privacy model training method, medium and system based on low-rank projection. Background Technology

[0002] Differential privacy provides a quantifiable, composable, and auditable privacy protection paradigm. Its intuitive meaning is that the distribution of the algorithm's output is "approximately consistent" regardless of whether a specific sample participates in the training, making it difficult for any external observer to reliably determine whether the sample participated or not.

[0003] In related technologies, when using isotropic Gaussian noise during differential privacy model training, the noise energy is evenly distributed across m directions. Since the actual effective learning signal is concentrated in a subspace much smaller than m, this "dimensional averaging" allocation strategy consumes a significant proportion of the budget in information-poor directions, resulting in slower early convergence and reduced final accuracy under the same privacy budget. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in related technologies. To this end, one objective of this invention is to propose a privacy model training method based on low-rank projection, which can effectively improve the convergence speed and training efficiency of the privacy model during training; simultaneously, it improves the accuracy of the final model prediction results.

[0005] In a first aspect, embodiments of the present invention propose a privacy model training method based on low-rank projection, comprising the following steps: sampling according to a preset sampling rate to obtain a corresponding sampling batch, and calculating the gradient corresponding to each sample in the sampling batch to obtain a gradient set; obtaining a periodic index parameter, and determining whether the current period has been completed based on the periodic index parameter; if the current period has been completed, estimating the covariance matrix based on the gradient set, and extracting the principal direction based on the covariance matrix and the projection rank to obtain an orthogonal basis, and calculating the corresponding energy retention ratio based on the orthogonal basis, and determining the principal subspace based on the energy retention ratio; calculating the projection threshold based on the energy retention ratio; calculating the average gradient corresponding to the gradient set, and projecting the average gradient onto the principal subspace to obtain the corresponding projection gradient; generating full-dimensional noise based on a preset noise multiplier and the projection threshold, and combining the projection gradient and the full-dimensional noise to generate a combined noise gradient, and updating the model parameters based on the combined noise gradient.

[0006] According to an embodiment of the present invention, a privacy model training method based on low-rank projection firstly samples according to a preset sampling rate to obtain a corresponding sampling batch, and calculates the gradient corresponding to each sample in the sampling batch to obtain a gradient set; then, obtains a periodic index parameter, and determines whether the current period has been completed based on the periodic index parameter; then, if the current period has been completed, estimates the covariance matrix based on the gradient set, extracts the principal direction based on the covariance matrix and the projection rank to obtain an orthogonal basis, and calculates the corresponding energy retention ratio based on the orthogonal basis to determine the principal subspace based on the energy retention ratio; then, calculates the projection threshold based on the energy retention ratio; then, calculates the average gradient corresponding to the gradient set, and projects the average gradient onto the principal subspace to obtain the corresponding projected gradient; then, generates full-dimensional noise based on a preset noise multiplier and the projection threshold, and combines the projected gradient and the full-dimensional noise to generate a combined noise gradient, and updates the model parameters based on the combined noise gradient. This effectively improves the convergence speed and training efficiency of the privacy model during training, and simultaneously improves the accuracy of the final model prediction results.

[0007] In some embodiments, after calculating the gradient corresponding to each sample in the sampling batch, the gradient is further clipped based on the following formula:

[0008]

[0009] in, This represents the gradient after clipping. This represents the gradient before clipping. The magnitude of the gradient vector is represented by the gradient vector. This represents the threshold.

[0010] In some embodiments, calculating the corresponding energy retention ratio based on the orthogonal basis includes: calculating the sum of the first eigenvalues ​​corresponding to the orthogonal basis and calculating the sum of the second eigenvalues ​​corresponding to the gradient set; calculating the ratio between the sum of the first eigenvalues ​​and the sum of the second eigenvalues, and using the ratio as the energy retention ratio.

[0011] In some embodiments, generating full-dimensional noise based on a preset noise multiplier and the projection threshold includes: generating Gaussian random vector pairs in the principal subspace based on the preset noise multiplier and the projection threshold; and mapping the Gaussian random vector pairs back to the full space to generate the full-dimensional noise.

[0012] In some embodiments, the combined noise gradient is generated according to the following formula:

[0013]

[0014] in, Represents the combined noise gradient. Represents the projected gradient. This represents full-dimensional noise.

[0015] In some embodiments, the method further includes: recording the privacy overhead of this step according to fixed parameters, and generating a privacy report from the RDP ledger after the number of training steps reaches a preset number of steps.

[0016] Secondly, embodiments of the present invention provide a computer-readable storage medium storing a privacy model training program based on low-rank projection, which, when executed by a processor, implements the privacy model training method based on low-rank projection as described above.

[0017] Thirdly, this invention proposes a privacy model training system based on low-rank projection, comprising: a sampling module, which performs sampling according to a preset sampling rate to obtain a corresponding sampling batch, and calculates the gradient corresponding to each sample in the sampling batch to obtain a gradient set; a judgment module, which obtains a periodic index parameter and determines whether the current period has been completed based on the periodic index parameter; and an update module, which estimates the covariance matrix based on the gradient set when the current period has been completed, and extracts the principal direction based on the covariance matrix and the projection rank to obtain an orthogonal basis. The system includes a calculation module for calculating the energy retention ratio based on the orthogonal basis, and a projection module for calculating the projected threshold based on the energy retention ratio; a projection module for calculating the average gradient corresponding to the gradient set and projecting the average gradient onto the principal subspace to obtain the corresponding projected gradient; and a training module for generating full-dimensional noise based on a preset noise multiplier and the projected threshold, combining the projected gradient and the full-dimensional noise to generate a combined noise gradient, and updating the model parameters based on the combined noise gradient.

[0018] In some embodiments, after calculating the gradient corresponding to each sample in the sampling batch, the gradient is further clipped based on the following formula:

[0019]

[0020] in, This represents the gradient after clipping. This represents the gradient before clipping. The magnitude of the gradient vector is represented by the gradient vector. This represents the threshold.

[0021] In some embodiments, calculating the corresponding energy retention ratio based on the orthogonal basis includes: calculating the sum of the first eigenvalues ​​corresponding to the orthogonal basis and calculating the sum of the second eigenvalues ​​corresponding to the gradient set; calculating the ratio between the sum of the first eigenvalues ​​and the sum of the second eigenvalues, and using the ratio as the energy retention ratio.

[0022] According to the privacy model training method based on low-rank projection of the present invention, the covariance matrix is ​​estimated based on the gradient set, and an orthogonal basis is calculated based on the covariance matrix and the projection rank, and the corresponding energy retention ratio is calculated based on the orthogonal basis; then, the projection threshold is dynamically adjusted based on the energy retention ratio; in this way, the privacy budget can be concentrated on the "effective direction", avoiding excessive pruning in the weak information direction to forcibly reduce sensitivity; improving the convergence speed and efficiency of model training; and improving the accuracy of the final model output.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the privacy model training method based on low-rank projection according to an embodiment of the present invention.

[0025] Figure 2 This is a block diagram of a privacy model training system based on low-rank projection according to an embodiment of the present invention. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0027] The following describes a privacy model training method based on low-rank projection according to an embodiment of the present invention with reference to the accompanying drawings.

[0028] Please see Figure 1 , Figure 1 This is a flowchart illustrating the privacy model training method based on low-rank projection according to an embodiment of the present invention. Figure 1 As shown, the privacy model training method based on low-rank projection includes the following steps:

[0029] S101, sample according to the preset sampling rate to obtain the corresponding sampling batch, and calculate the gradient corresponding to each sample in the sampling batch to obtain the gradient set.

[0030] In some embodiments, after calculating the gradient corresponding to each sample in the sampling batch, the gradient is further clipped based on the following formula:

[0031]

[0032] in, This represents the gradient after clipping. This represents the gradient before clipping. The magnitude of the gradient vector is represented by the gradient vector. This represents the threshold.

[0033] As an example, firstly, sampling is performed according to a preset sampling rate to obtain the corresponding sampling batch; then, for each sample in the sampling batch, its corresponding gradient is calculated. Then, perform gradient clipping:

[0034]

[0035] in, This represents the gradient after clipping. This represents the gradient before clipping. The magnitude of the gradient vector is represented by the gradient vector. This represents the threshold.

[0036] Then, based on the gradients after clipping each sample, a gradient set can be obtained. It can be understood that by clipping the gradients, the influence of a single sample can have a unified upper bound.

[0037] S102, obtain the periodic index parameter, and determine whether the current period has been completed based on the periodic index parameter.

[0038] S103, if the current cycle has been completed, estimate the covariance matrix based on the gradient set, extract the principal direction based on the covariance matrix and the projected rank to obtain the orthogonal basis, and calculate the corresponding energy retention ratio based on the orthogonal basis to determine the principal subspace based on the energy retention ratio.

[0039] S104, calculate the projected threshold based on the energy retention ratio.

[0040] In some embodiments, calculating the corresponding energy retention ratio based on the orthogonal basis includes: calculating the sum of the first eigenvalues ​​corresponding to the orthogonal basis and calculating the sum of the second eigenvalues ​​corresponding to the gradient set; calculating the ratio between the sum of the first eigenvalues ​​and the sum of the second eigenvalues, and using the ratio as the energy retention ratio.

[0041] In other words, after obtaining the gradient set corresponding to the current step's data, the periodic index parameter is used to determine whether the current cycle has been completed after the current step's data acquisition is finished. If so, the covariance matrix is ​​estimated based on the gradient set, and the principal directions are extracted based on the covariance matrix and the projected rank to obtain an orthogonal basis. The corresponding energy retention ratio is then calculated based on the orthogonal basis to determine the principal subspace. This completes the update of the principal subspace. Simultaneously, the projected threshold is calculated using the energy retention ratio, and the projected threshold is adjusted accordingly. If the current cycle is not completed, there is no need to update the principal subspace or adjust the projected threshold; the process directly proceeds to calculating the average gradient corresponding to the gradient set.

[0042] As an example, first, determine whether the current period has been completed based on the periodic index parameter; if so, then base the gradient set on the current batch or sliding window { Estimate the covariance matrix Σg; then, extract its first r′ (projected rank) principal directions using the SVD / randomized SVD / EMA method to form an orthogonal basis. ;in, Represents the set of real numbers. Let m represent the set of all real matrices with m rows and r′ columns; m represents the dimension of the LoRA gradient vectors, and r′ represents the projection rank, i.e., the number of principal directions retained. Therefore, Let U be an orthogonal basis matrix consisting of r′ principal direction vectors of length m. This allows for the extraction of "strong information directions," enabling subsequent injection of noise into truly meaningful directions. Then, the energy retention ratio is calculated. ∈ [0, 1] (defined as the ratio of the sum of the first r′ singular values ​​to the sum of all singular values). Then, the corresponding projected threshold is calculated based on this energy retention ratio:

[0043]

[0044] in, Indicates the threshold after projection. Indicates the energy retention ratio. This represents the original threshold.

[0045] It is understandable that dynamically adjusting the projection threshold based on the energy retention ratio is equivalent to recalibrating the upper bound of sensitivity at the scale of the "effective subspace". This can avoid reserving too much budget for weak information directions, improve the convergence speed, and at the same time, improve the accuracy of the model prediction results.

[0046] S105, calculate the average gradient corresponding to the gradient set, and project the average gradient onto the principal subspace to obtain the corresponding projected gradient.

[0047] As an example, first, calculate the average gradient:

[0048]

[0049] in, Represents the average gradient. Indicates the size of the current training batch. Indicates the first The clipped gradient corresponding to each sample.

[0050] Next, the average gradient is projected onto the principal subspace to obtain the projected gradient:

[0051]

[0052] in, Represents the projected gradient. Represents the projected rank. Describes an orthogonal basis matrix. Represent its transpose matrix; Represents the average gradient. Represents the projected rank The orthogonal projection is used to project the average gradient onto the plane formed by the previous... In the principal subspace formed by the principal directions, it can be represented as .in, Describes an orthogonal basis matrix. This represents its transpose matrix.

[0053] It should be noted that this method allows the projected gradient to retain the main learning signal, significantly reducing the dimensionality burden that requires subsequent noise injection.

[0054] S106 generates full-dimensional noise based on a preset noise multiplier and a projection threshold, and combines the projection gradient and full-dimensional noise to generate a combined noise gradient, and updates the model parameters based on the combined noise gradient.

[0055] In some embodiments, generating full-dimensional noise based on a preset noise multiplier and the projection threshold includes: generating Gaussian random vector pairs in the principal subspace based on the preset noise multiplier and the projection threshold; and mapping the Gaussian random vector pairs back to the full space to generate the full-dimensional noise.

[0056] In some embodiments, the combined noise gradient is generated according to the following formula:

[0057]

[0058] in, Represents the combined noise gradient. Represents the projected gradient. This represents full-dimensional noise.

[0059] As an example, firstly, in R r′ Internal Press Generate negatively correlated Gaussian vector pairs ;in, Indicates the preset noise level. Represents the correlation coefficient. This represents the projected threshold; specifically, it can be implemented using a linear combination of a shared standard Gaussian vector and independent compensation vectors. Then, the block diagonal mapping diag(U, U) is used to... Back-mapping ,and Segmentation alignment:

[0060]

[0061] Then, by passing the combined noise gradient to the optimizer, the model parameters can be updated.

[0062] In some embodiments, the method further includes: recording the privacy overhead of this step according to fixed parameters, and generating a privacy report from the RDP ledger after the number of training steps reaches a preset number of steps.

[0063] As an example, firstly, the privacy overhead of this step is recorded with fixed parameters; then, after the number of training steps reaches a preset threshold, the final privacy report is generated from the RDP ledger; verification. Solidify the deliverables (model weights, logs, ledgers, configuration lists) for audit recalculation.

[0064] It should be noted that during the deployment and inference phases, only W0 and the LoRA factor (A, B) need to be loaded during inference; pruning, projection, and noise addition only occur during the training phase, and the deployment and inference paths do not require modification. Regarding operation and maintenance, if B or E needs to be changed (thus altering q or T), a new search must be performed. If r′ needs to be adjusted, or T proj It is possible to do so without changing (q, Under the premise of T), it takes effect directly; logs and audit information must be recorded throughout the entire process.

[0065] In summary, the privacy model training method based on low-rank projection according to embodiments of the present invention firstly involves sampling according to a preset sampling rate to obtain a corresponding sampling batch, and calculating the gradient corresponding to each sample in the sampling batch to obtain a gradient set; then, obtaining a periodic index parameter, and determining whether the current period has been completed based on the periodic index parameter; then, if the current period has been completed, estimating the covariance matrix based on the gradient set, extracting the principal direction based on the covariance matrix and the projection rank to obtain an orthogonal basis, and calculating the corresponding energy retention ratio based on the orthogonal basis to determine the principal subspace based on the energy retention ratio; next, calculating the projection threshold based on the energy retention ratio; then, calculating the average gradient corresponding to the gradient set, and projecting the average gradient onto the principal subspace to obtain the corresponding projected gradient; next, generating full-dimensional noise based on a preset noise multiplier and the projection threshold, and combining the projected gradient and the full-dimensional noise to generate a combined noise gradient, and updating the model parameters based on the combined noise gradient. This effectively improves the convergence speed and training efficiency of privacy models during training, while also increasing the accuracy of the final model prediction results.

[0066] Secondly, embodiments of the present invention provide a computer-readable storage medium storing a privacy model training program based on low-rank projection, which, when executed by a processor, implements the privacy model training method based on low-rank projection as described above.

[0067] Thirdly, embodiments of the present invention propose a privacy model training system based on low-rank projection, which includes: a sampling module 10, a judgment module 20, an update module 30, a calculation module 40, a projection module 50, and a training module 60.

[0068] The sampling module 10 is used to sample according to a preset sampling rate to obtain a corresponding sampling batch, and to calculate the gradient corresponding to each sample in the sampling batch to obtain a gradient set.

[0069] The judgment module 20 is used to obtain the periodic index parameter and determine whether the current period has been completed based on the periodic index parameter;

[0070] The update module 30 is used to estimate the covariance matrix based on the gradient set when the current cycle has been completed, extract the principal direction based on the covariance matrix and the projected rank to obtain the orthogonal basis, and calculate the corresponding energy retention ratio based on the orthogonal basis to determine the principal subspace based on the energy retention ratio.

[0071] Calculation module 40 is used to calculate the projected threshold based on the energy retention ratio;

[0072] The projection module 50 is used to calculate the average gradient corresponding to the gradient set and project the average gradient onto the principal subspace to obtain the corresponding projected gradient.

[0073] The training module 60 is used to generate full-dimensional noise based on a preset noise multiplier and a projection threshold, and to combine the projection gradient and the full-dimensional noise to generate a combined noise gradient, and to update the model parameters based on the combined noise gradient.

[0074] In some embodiments, after calculating the gradient corresponding to each sample in the sampling batch, the gradient is further clipped based on the following formula:

[0075]

[0076] in, This represents the gradient after clipping. This represents the gradient before clipping. The magnitude of the gradient vector is represented by the gradient vector. This represents the threshold.

[0077] In some embodiments, calculating the corresponding energy retention ratio based on the orthogonal basis includes: calculating the sum of the first eigenvalues ​​corresponding to the orthogonal basis and calculating the sum of the second eigenvalues ​​corresponding to the gradient set; calculating the ratio between the sum of the first eigenvalues ​​and the sum of the second eigenvalues, and using the ratio as the energy retention ratio.

[0078] It should be noted that the above description of the privacy model training method based on low-rank projection also applies to this privacy model training system based on low-rank projection, and will not be repeated here.

[0079] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0080] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0081] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0082] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0084] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0085] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0086] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for training a privacy model based on low-rank projection, characterized in that, Includes the following steps: Sampling is performed according to a preset sampling rate to obtain a corresponding sampling batch, and the gradient corresponding to each sample in the sampling batch is calculated to obtain a gradient set; Obtain the periodic index parameters, and determine whether the current period has been completed based on the periodic index parameters; If the current cycle has been completed, the covariance matrix is ​​estimated based on the gradient set, and the principal direction is extracted based on the covariance matrix and the projected rank to obtain an orthogonal basis. The corresponding energy retention ratio is calculated based on the orthogonal basis, and the principal subspace is determined based on the energy retention ratio. Calculate the projected threshold based on the energy retention ratio; Calculate the average gradient corresponding to the gradient set, and project the average gradient onto the principal subspace to obtain the corresponding projected gradient; Full-dimensional noise is generated based on a preset noise multiplier and the projection threshold, and the projection gradient and the full-dimensional noise are combined to generate a combined noise gradient, and the model parameters are updated based on the combined noise gradient. The calculation of the corresponding energy retention ratio based on the orthogonal basis includes: Calculate the sum of the first eigenvalues ​​corresponding to the orthogonal basis, and calculate the sum of the second eigenvalues ​​corresponding to the gradient set; Calculate the ratio between the sum of the first eigenvalues ​​and the sum of the second eigenvalues, and use the ratio as the energy retention ratio.

2. The privacy model training method based on low-rank projection as described in claim 1, characterized in that, After calculating the gradient for each sample in the sampling batch, the gradient is further clipped based on the following formula: in, Indicates the first The clipped gradient for each sample Indicates the first The gradient before clipping for each sample. The magnitude of the gradient vector is represented by the gradient vector. This represents the threshold.

3. The privacy model training method based on low-rank projection as described in claim 1, characterized in that, Full-dimensional noise is generated based on a preset noise factor and the projected threshold, including: Based on the preset noise ratio and the projection threshold, Gaussian random vector pairs are generated in the principal subspace; The Gaussian random vector is back-mapped to the full space to generate the full-dimensional noise.

4. The privacy model training method based on low-rank projection as described in claim 1, characterized in that, The combined noise gradient is generated according to the following formula: in, Represents the combined noise gradient. Represents the projected gradient. This represents full-dimensional noise.

5. The privacy model training method based on low-rank projection as described in claim 1, characterized in that, Also includes: The privacy overhead of this step is recorded based on fixed parameters, and a privacy report is generated by the RDP ledger after the number of training steps reaches a preset number.

6. A computer-readable storage medium, characterized in that, It stores a privacy model training program based on low-rank projection, which, when executed by the processor, implements the privacy model training method based on low-rank projection as described in any one of claims 1-5.

7. A privacy model training system based on low-rank projection, characterized in that, include: A sampling module is used to sample according to a preset sampling rate to obtain a corresponding sampling batch, and to calculate the gradient corresponding to each sample in the sampling batch to obtain a gradient set. The judgment module is used to obtain periodic index parameters and determine whether the current period has been completed based on the periodic index parameters. The update module is used to estimate the covariance matrix based on the gradient set when the current cycle has been completed, extract the principal direction according to the covariance matrix and the projection rank to obtain the orthogonal basis, and calculate the corresponding energy retention ratio according to the orthogonal basis to determine the principal subspace according to the energy retention ratio. The calculation module is used to calculate the projected threshold based on the energy retention ratio; A projection module is used to calculate the average gradient corresponding to the gradient set and project the average gradient onto the principal subspace to obtain the corresponding projected gradient. The training module is used to generate full-dimensional noise based on a preset noise multiplier and the projection threshold, and to combine the projection gradient and the full-dimensional noise to generate a combined noise gradient, and to update the model parameters according to the combined noise gradient. The calculation of the corresponding energy retention ratio based on the orthogonal basis includes: Calculate the sum of the first eigenvalues ​​corresponding to the orthogonal basis, and calculate the sum of the second eigenvalues ​​corresponding to the gradient set; Calculate the ratio between the sum of the first eigenvalues ​​and the sum of the second eigenvalues, and use the ratio as the energy retention ratio.

8. The privacy model training system based on low-rank projection as described in claim 7, characterized in that, After calculating the gradient for each sample in the sampling batch, the gradient is further clipped based on the following formula: in, This represents the gradient after clipping. This represents the gradient before clipping. The magnitude of the gradient vector is represented by the gradient vector. This represents the threshold.

Images