Influence function decomposition-based large model privacy knowledge decoupling forgetting method and system

Abstract

The present application relates to the technical field of privacy protection in large language model application, and discloses a large model privacy knowledge decoupling forgetting method and system based on influence function decomposition. The method obtains a privacy sample set and a knowledge reservation set; a random orthogonal projection matrix is constructed, the Hessian matrix of the large language model parameter space is calculated and projected to a low-dimensional subspace, and the influence function of the samples in the space set is used to quantify the influence on the model parameters after removing the samples; the principal components of the influence function of the knowledge reservation set are extracted, and a knowledge subspace base is constructed; the influence function of the privacy sample is projected to the knowledge subspace base, and the knowledge component and the privacy component are obtained by decomposition; a parameter correction strategy is designed accordingly, and the model parameters are updated according to the strategy, so that the privacy influence is eliminated along the privacy component direction while the knowledge loss is compensated along the knowledge component direction, thereby iteratively optimizing the model. The present application realizes the optimal balance between sufficient forgetting of privacy data and preservation of model general ability.

Description

Technical Field

[0001] This invention relates to the field of privacy protection technology in large language model applications, specifically to a method and system for decoupling and forgetting privacy knowledge in large models based on influence function decomposition. Background Technology

[0002] Large-scale pre-trained language models have demonstrated powerful capabilities in natural language processing, code generation, and the application of specialized domain knowledge. By pre-training and fine-tuning on massive amounts of text data, large language models can encode human knowledge in a distributed manner within neural networks with billions to hundreds of billions of parameters. However, the training of large models inevitably involves sensitive data and privacy information, potentially leaking sensitive information from the training set, such as personal identification information, medical records, financial records, and proprietary code, through specific prompts. Simultaneously, privacy attack techniques targeting large models, such as member inference attacks, model reverse engineering, and prompt injection, are constantly evolving, posing a serious challenge to data privacy protection.

[0003] Machine forgetting technology aims to enable trained models to selectively forget the impact of specific data, and is a key technical means to address the demands for data privacy protection and the right to be forgotten. Current forgetting methods for large models mainly include: full retraining, which retrains the model from the remaining dataset, fundamentally eliminating the impact of the target data, but requires weeks to months of training time and enormous computational costs for models with hundreds of billions of parameters; gradient ascent, which maximizes the loss function of the target data by performing backward gradient updates on the target data to reduce the model's fit to it; this method is computationally efficient, but fine-grained control of the forgetting range and intensity remains a challenge; and parameter localization and deletion, which achieves forgetting by identifying and zeroing or pruning the subset of parameters encoding the target knowledge. Because the knowledge in large models exhibits highly distributed storage characteristics, individual knowledge points are often scattered across multiple layers and parameters, requiring complex analytical techniques for precise localization.

[0004] Influence function techniques originate from robust statistics. By calculating the marginal contribution of training samples to model parameters, they can quantify the direction and magnitude of changes in model parameters after the removal of a single sample. The core idea of ​​this technique is to use influence functions to characterize the impact of samples on model parameters. Influence function techniques have already been applied in data attribution, outlier detection, and training set selection in traditional machine learning models. Recent research attempts to introduce influence functions into deep learning scenarios, identifying training samples that contribute the most to specific predictions through influence scores, thus supporting model interpretability and data source attribution. Theoretically, influence functions can accurately characterize the causal relationship between samples and parameters, providing an ideal technical path for precise forgetting. However, classic influence functions rely on the calculation and inversion of the Hessian matrix, which faces the curse of dimensionality problem in the high-dimensional parameter space of large models, making influence functions highly complex in large-scale model scenarios. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention provides a method and system for decoupling and forgetting privacy knowledge in large models based on influence function decomposition. This method can efficiently quantify the feasibility of evaluating the impact of training samples on the parameters of large language models and achieves the optimal balance between sufficient forgetting of privacy data and preservation of the model's generality.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention discloses a method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition, comprising:

[0008] S1. Obtain the set of privacy samples to be forgotten and the knowledge retention set for maintaining general capabilities;

[0009] S2. Construct a random orthogonal projection matrix, and project the Hessian matrix of the large language model parameter space to a low-dimensional subspace through the projection matrix. Solve for the Hessian inverse matrix in the low-dimensional subspace, and calculate the influence function of the privacy sample set and the samples in the knowledge retention set based on the inverse matrix, so as to quantify the impact of removing samples on the model parameters.

[0010] S3. Principal component extraction is performed on the influence function of the knowledge retention set through singular value decomposition to construct a knowledge subspace basis; the influence function of the privacy sample is projected onto the knowledge subspace basis and orthogonally decomposed to obtain knowledge components and privacy components; wherein, the knowledge component represents the part of the influence of the privacy sample on the model parameters that is consistent with the direction of the influence of the parameters of the knowledge retention set; the privacy component represents the part of the influence of the privacy sample on the model parameters that is orthogonal to the direction of the influence of the parameters of the knowledge retention set.

[0011] S4. Design a parameter correction strategy based on the knowledge component and the privacy component, and update the model parameters according to the parameter correction strategy. While eliminating the privacy impact along the privacy component direction, compensate for knowledge loss along the knowledge component direction, thereby iteratively optimizing the large language model.

[0012] As a further improvement to the above scheme, step S2 specifically includes:

[0013] S21. Generate a random orthogonal projection matrix :

[0014] ;

[0015] In the formula, , This represents the total number of model parameters. Let be the subspace dimension, and Less than , Let be the set of real numbers, satisfying superscript It is the transpose symbol. for An identity matrix of order 1; This indicates that after performing QR decomposition on the matrix, only the orthogonal matrix Q is taken; This indicates a range from a mean of 0 and a standard deviation of 0. Sampling in Gaussian distribution The elements constitute the initial matrix;

[0016] S22. Divide the training set used to train the large language model into... The batch number, recorded as the [number]th batch. The average Hessian matrix of each batch of samples is: ;in ;

[0017] S23. Project the matrix Decomposed into columns Calculate the Hessian-vector product for each column separately. spliced ​​together Then calculate each batch Projection results Finally, for The low-dimensional subspace is obtained by summing and averaging the projection results of each batch. ;in ;

[0018] S24. Solving for the regularized inverse matrix in low-dimensional space , ;in The damping coefficient;

[0019] S25. For the target samples in the privacy sample set or the knowledge retention set. The influence function is calculated using the following formula:

[0020] ;

[0021] In the formula, For the sample Influence function; For target samples In optimal parameters The loss gradient at that point.

[0022] As a further improvement to the above scheme, in step S2, the computational efficiency of the influence function is improved in the following ways:

[0023] Based on model characteristics, it is divided into Each layer calculates its influence function independently; the hierarchical influence function decomposition is expressed as:

[0024] ;

[0025] In the formula, the symbol This indicates vector concatenation; For layer index; For target samples In the model The influence function of the layer is calculated using the following formula:

[0026] ;

[0027] In the formula, For the model number Layer parameters, This is the projection matrix of this layer; For target samples In the optimal parameters of the model The loss function at point 1 for the first point 2 Layer parameters The gradient vector;

[0028] In the case of batch forgetting multiple samples, the first sample is executed completely. , and The calculation is performed so that subsequent samples only calculate the new gradient and reuse the previously calculated gradient. and .

[0029] As a further improvement to the above scheme, step S3 specifically includes:

[0030] S31. Obtain the set of influence functions for all samples in the knowledge retention set: ;in For knowledge reservation;

[0031] S32. Extraction via singular value decomposition Main direction of influence: ;in Represents the singular value decomposition function; It is a left singular vector matrix, that is, orthogonal to the parameter space; It is a singular value diagonal matrix; It is a right singular vector matrix;

[0032] S33. From Take the middle before Each principal component constitutes a knowledge subspace basis: ;

[0033] S34. Regarding privacy samples The influence function is decomposed into knowledge components. and privacy ;in, , ; It is an identity matrix.

[0034] As a further improvement to the above scheme, step S4 specifically includes:

[0035] S41. Calculate the ratio of the magnitude of the privacy component to the magnitude of the knowledge component, and allocate each privacy sample according to the ratio. It is divided into three levels: high risk, medium risk, and low risk.

[0036] S42. Calculate the privacy impact of each layer of the model using the following formula:

[0037] ;

[0038] In the formula, For the first A forgotten privacy sample In the model The impact of layered privacy; For the first Privacy components of a privacy sample In the Layer component vectors; aggregated hierarchical privacy impact The larger the value, the higher the value of the model. The stronger the encoding of privacy information in each layer; It is an L2 norm;

[0039] S43. Identify the privacy-critical layers based on the privacy impact of each layer, i.e., the layers that need to be prioritized and forgotten:

[0040] ;

[0041] In the formula, For privacy-critical layers; Privacy impact score after traversing all L layers of the model The maximum value is used as the benchmark threshold for determining the privacy-critical layer;

[0042] S44. Use a differentiated update strategy for the parameters of different layers of the model:

[0043] ;

[0044] In the formula, The total number of privacy samples to be forgotten; For the first Forgetting weights for privacy samples and These are the privacy correction coefficient and the knowledge compensation coefficient, all of which are determined based on the risk level of the privacy sample. For the model number The original optimal parameters of the layer; For the model after the forgetting operation New parameters for the layer; For the first The corrected strength of the layer; For the first The knowledge components of a privacy sample in the th Layer component vectors.

[0045] As a further improvement to the above scheme, in step S41, if If the privacy sample is high-risk, then the sample forgetting weight is applied. Privacy correction factor Knowledge compensation coefficient ;

[0046] like If the privacy sample is considered medium risk, then the sample forgetting weight is... Privacy correction factor Knowledge compensation coefficient ;

[0047] like If the privacy sample is low risk, then the sample forgetting weight is applied. Privacy correction factor Knowledge compensation coefficient .

[0048] As a further improvement to the above scheme, a global parameter offset constraint is also introduced in step S44:

[0049] Define the regularization loss for parameter offsets. The constraint is: the parameter correction should satisfy... ;in This means that the overall variation in parameters is allowed to be no more than 1%;

[0050] If the parameter adjustment exceeds the constraint, adaptive scaling is performed, as shown in the following expression:

[0051] ;

[0052] In the formula, This is the scaling factor. ; To obtain the minimum value.

[0053] As a further improvement to the above scheme, in step S4, if the total number of privacy samples to be forgotten... If the threshold is exceeded, multiple rounds of iterative optimization and updates will be used:

[0054] The privacy samples are divided into multiple batches and forgotten sequentially according to batch order. In each batch of each iteration, the privacy and knowledge components of the sample are calculated based on the current parameters, and parameter correction and regularization constraints are applied. The privacy correction coefficient is dynamically decayed as the iteration progresses. And increase the knowledge compensation coefficient The iteration terminates when the privacy loss threshold, maximum number of iterations, or loss convergence condition is met.

[0055] As a further improvement to the above scheme, in step S1, the knowledge retention set contains three types of samples: the first type is non-privacy samples of the same task, which belong to the same task type as the privacy samples but do not contain sensitive information; the second type is general domain knowledge samples, which contain encyclopedic knowledge or professional guides in related fields; the third type is domain generalization samples, which are used to evaluate the general reasoning ability of the model; wherein, the knowledge retention set is constructed to overlap with the privacy sample set in terms of domain knowledge features, but to be orthogonal to the privacy sample set in terms of privacy information features.

[0056] This invention also discloses a large-scale model privacy knowledge decoupling and forgetting system based on influence function decomposition, applying the large-scale model privacy knowledge decoupling and forgetting method based on influence function decomposition as described above; the system includes:

[0057] The influence function calculation module is used to obtain the privacy sample set to be forgotten and the knowledge retention set for maintaining general capabilities; construct a random orthogonal projection matrix to project the Hessian matrix of the large language model parameter space to a low-dimensional subspace; and calculate the influence function of the privacy sample set and the samples in the knowledge retention set based on the low-dimensional subspace to quantify the impact of removing samples on the model parameters.

[0058] The privacy knowledge decomposition module is used to extract principal components from the influence function of the knowledge retention set and construct a knowledge subspace basis; the influence function of the privacy sample is projected onto the knowledge subspace basis to decompose it into knowledge components and privacy components.

[0059] A selective forgetting execution module is used to design a parameter correction strategy based on the knowledge component and the privacy component;

[0060] The model parameter update module is used to design parameter correction strategies based on the knowledge components and the privacy components, and update the model parameters according to the parameter correction strategies. While eliminating the privacy impact along the privacy component direction, it compensates for knowledge loss along the knowledge component direction, thereby iteratively optimizing the large language model.

[0061] Compared with the prior art, the beneficial effects of the present invention are:

[0062] 1. While existing influence function methods can accurately quantify the impact of training samples on model parameters, their core relies on the calculation and inversion of the Hessian matrix, which faces enormous storage overhead and computational complexity in large language models with hundreds of billions of parameters. This invention introduces random orthogonal projection and low-rank subspace approximation techniques to project the Hessian operations in the high-dimensional parameter space into a low-dimensional subspace, reducing the space and time complexity of computation. This makes the influence function technique feasible for ultra-large-scale models, laying a computable quantitative foundation for subsequent efficient and accurate forgetting operations.

[0063] 2. Traditional machine forgetting methods generally employ a holistic forgetting strategy, applying indiscriminate gradient ascent or parameter perturbation to all parameters related to the data to be forgotten. This fails to distinguish between privacy-specific information and general knowledge components at the parameter level, inevitably damaging the model's general knowledge and reasoning ability acquired in the domain during privacy elimination, resulting in severe over-forgetting problems. This invention constructs an influence subspace comparative analysis mechanism to extract the differences in influence patterns between the data to be forgotten and the knowledge retention reference set in the parameter space. Using orthogonal projection decomposition technology, it precisely decomposes the parameter influence of each sample to be forgotten into privacy-specific and knowledge-sharing components, and classifies the risk based on their ratio. This achieves fine-grained separation of privacy information and general knowledge at the parameter influence level, providing clear parameter-level operational guidance for selective forgetting and significantly improving the accuracy of forgetting and the integrity of knowledge retention.

[0064] 3. Existing forgetting methods face a dilemma between thorough forgetting and model performance: aggressive forgetting strategies, while effectively eliminating privacy, can lead to a significant degradation or even complete failure of the model's generality; conservative forgetting strategies, while maintaining performance, pose a high risk of privacy residue and fail to meet regulatory requirements. This invention designs a risk-based bidirectional differentiated parameter correction strategy, innovatively employing a bidirectional adjustment mechanism of "strengthening forgetting along the privacy direction and actively compensating along the knowledge direction." Based on the privacy-knowledge decoupling results, the forgetting intensity coefficient and knowledge compensation coefficient of each parameter are dynamically determined. Combined with hierarchical selective correction and regularization constraints, this achieves precise targeting of high-risk privacy parameters and effective protection of shared knowledge parameters. This parameter-level fine-grained control strategy ensures sufficient privacy elimination while minimizing damage to the model's generality, making the forgetting operation more gentle, controllable, and efficient. Attached Figure Description

[0065] Figure 1 This is a flowchart of the privacy knowledge decoupling and forgetting method for large models based on influence function decomposition in Embodiment 1 of the present invention.

[0066] Figure 2 This is an architecture diagram of the large-model privacy knowledge decoupling forgetting method based on influence function decomposition in Embodiment 1 of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Example 1

[0069] Existing large-scale model forgetting techniques have the following main technical limitations:

[0070] (1) The computational feasibility of classical influence functions in large model scenarios

[0071] Influence function techniques can theoretically accurately characterize the causal impact of training samples on model parameters, providing an ideal path for precise forgetting. However, classic influence function methods require calculating and inverting the Hessian matrix in the model parameter space. For large language models containing billions of parameters, the dimensionality of the Hessian matrix explodes, resulting in high computational complexity and low feasibility in engineering practice. This makes influence function techniques difficult to play a role in real-world large-scale model forgetting tasks.

[0072] (2) The problem of fine-grained separation between privacy information and general knowledge

[0073] Training data typically contains both privacy-sensitive information and reusable general knowledge components. For example, financial consulting samples contain both client privacy information and general financial knowledge such as risk assessment methods and investment strategies; medical diagnostic samples contain both patient identifiable information and general medical knowledge such as the correlation between diseases and symptoms. Ideally, forgetting should selectively eliminate privacy-specific components while retaining general knowledge components to maintain model performance. However, existing forgetting methods lack mechanisms to distinguish between these two types of knowledge components at the parameter impact level, often adopting a holistic forgetting strategy for the target data, which can easily lead to excessive forgetting and significantly impair the model's generality.

[0074] To address this, this embodiment provides a large-scale model privacy knowledge decoupling and forgetting method based on influence function decomposition. This method overcomes the computational bottleneck of classical influence functions through low-rank subspace projection technology, constructing an efficient influence path tracing mechanism from training samples to model parameters. It employs orthogonal decomposition of influence functions to separate the parameter influence of the data to be forgotten into "privacy-specific components" and "knowledge-sharing components," achieving precise decoupling of privacy information from general knowledge. Based on the influence component analysis results, a risk-level differentiated parameter correction strategy is designed, employing adaptive forgetting intensity and knowledge compensation coefficients for samples with different privacy risk levels. Finally, through a multi-round iterative optimization framework with regularization constraints, the optimal balance between sufficient forgetting of privacy data and maintenance of the model's general capabilities is achieved.

[0075] Please see Figure 1 and Figure 2 The method includes steps S1 to S4.

[0076] S1. Obtain the privacy sample set to be forgotten. and knowledge retention sets used to maintain general capabilities .

[0077] The knowledge retention set contains three types of samples:

[0078] (1) Non-privacy samples of the same task, which belong to the same task type as privacy samples but do not contain sensitive information (such as personal information);

[0079] (2) General domain knowledge samples, including publicly available knowledge such as encyclopedic knowledge or professional guides in related fields;

[0080] (3) Domain generalization samples and other domain task samples are used to evaluate the general reasoning ability of the model.

[0081] in, and They overlap in domain knowledge but are orthogonal in privacy information.

[0082] S2. Construct a random orthogonal projection matrix, and project the Hessian matrix of the large language model parameter space to a low-dimensional subspace through the projection matrix. Solve for the Hessian inverse matrix in the low-dimensional subspace, and calculate the influence function of the privacy sample set and the samples in the knowledge retention set based on the inverse matrix, so as to quantify the impact of removing samples on the model parameters.

[0083] First, the theoretical basis for the influence function is as follows:

[0084] For parameters Large models, in the training set The optimal parameters are obtained through training. ,in For the total number of parameters, For the set of real numbers, Indicates the first training samples, For input text, is the corresponding label, and N is the total number of training samples.

[0085] The influence function is used to quantize the removal of training samples. The subsequent impact on model parameters:

[0086] ;

[0087] in, For the sample The loss gradient, i.e., the loss function. The first derivative with respect to the parameter; To train the loss Hessian matrix, loss function Second-order partial derivatives with respect to the parameters. It is the inverse of the Hessian matrix, used to convert the gradient into the actual change in the parameter space.

[0088] For a model with hundreds of billions of parameters ( The space complexity of calculating the Hessian matrix is ​​O(n). Direct computation is difficult. Therefore, this invention employs a random projection strategy to project the Hessian computation onto a low-dimensional subspace, utilizing the low-rank property of the loss surface to achieve an efficient approximation.

[0089] Step S2 of the present invention specifically includes S21 to S25.

[0090] S21. Generate a random orthogonal projection matrix :

[0091] ;

[0092] In the formula, , This represents the total number of model parameters. Let be the subspace dimension, and Less than , Let be the set of real numbers, satisfying superscript It is the transpose symbol. for An identity matrix of order 1; This indicates that after performing QR decomposition on the matrix, only the orthogonal matrix Q is taken; This indicates a range from a mean of 0 and a standard deviation of 0. Sampling in Gaussian distribution The elements constitute the initial matrix. In this embodiment, the subspace dimension... Determined adaptively based on model size: , .

[0093] Then, the complete Hessian can be projected onto a lower-dimensional subspace:

[0094]

[0095] By matrix multiplication, the original Hessian dimensionality reduction to dimension.

[0096] To avoid explicitly building the complete A 3D Hessian matrix is ​​used, employing the Hessian-vector product technique to avoid explicitly constructing the complete Hessian matrix, by dividing the training set into 3D parts. In each batch, leveraging the additivity of Hessian, the projection results of each batch are iteratively accumulated:

[0097] ;

[0098] in, This represents the subset of samples included in the b-th batch. Let b be the average loss of the b-th batch of samples. Let be the average Hessian matrix of the b-th batch of samples.

[0099] However, even for a single batch... The 3D Hessian matrix still cannot be explicitly constructed. The key observation is the computation... Only the product of the Hessian and the projection matrix is ​​needed; the complete Hessian is not required. Let the Hessian of the b-th batch be denoted as . , needs to be calculated Therefore, there are steps S22~S23.

[0100] S22. Divide the training set used to train the large language model into... The batch number, recorded as the [number]th batch. The average Hessian matrix of each batch of samples is: ;in .

[0101] S23. Project the matrix Decomposed into columns Calculate the Hessian-vector product for each column separately. spliced ​​together Then calculate each batch Projection results Finally, for The low-dimensional subspace is obtained by summing and averaging the projection results of each batch. ;in The Hessian-vector product can be calculated using the second derivative of automatic differentiation, without needing to construct a complete Hessian:

[0102] ;

[0103] in This is the first-order gradient of the batch loss. The calculation process is as follows: first calculate the first-order gradient. Then calculate the scalar. gradient with respect to parameters The space complexity of a single Hessian-vector product is O(n).

[0104] The complete calculation process is as follows: for each batch Calculate the k-th Hessian-vector product to obtain , and then calculate Finally, for The average results of each batch were obtained .

[0105] S24. Solving for the regularized inverse matrix in low-dimensional space , ;in This is the damping coefficient. In this embodiment, , , for An identity matrix of order 1. For matrix The trace is the sum of the diagonal elements. Because The Cholesky decomposition is used to directly find the inverse.

[0106] S25. For the target samples in the privacy sample set or the knowledge retention set. The influence function is approximated using the following formula:

[0107] ;

[0108] In the formula, For the sample Influence function; For target samples In optimal parameters The loss gradient at that point.

[0109] The calculation process is as follows: Calculate the gradient of the target sample. Projected onto subspace Subspace solution Project back to full parameter space .

[0110] In step S2, the computational efficiency of the influence function is improved in the following ways:

[0111] Based on model characteristics, it is divided into Each layer calculates its influence function independently; the hierarchical influence function decomposition is expressed as:

[0112] ;

[0113] In the formula, the symbol This indicates vector concatenation; For layer index; For target samples In the model The influence function of the layer is calculated using the following formula:

[0114] ;

[0115] In the formula, For the model number Layer parameters, The projection matrix of this layer ( This represents the number of parameters in this layer. (where the dimension of this subspace is...) For target samples In the optimal parameters of the model The loss function at point 1 for the first point 2 Layer parameters The gradient vector.

[0116] Wherein, the projection matrix P, the subspace Hessian Inverse matrix The calculations depend only on the model parameters and the training set, and are independent of the specific samples to be forgotten. The computation of these matrices is the most time-consuming part of the entire process.

[0117] For the first sample: Execute P completely. , Calculation;

[0118] For subsequent samples: directly reuse the already calculated P and Only the gradient specific to this sample needs to be calculated. Substitute into the formula That's all.

[0119] This batch processing of K samples avoids K repeated calculations of the Hessian inverse matrix, significantly improving efficiency.

[0120] Compared to the space complexity of direct methods This method only requires a projection matrix. ( Intermediate results ( ) and subspace Hessian ( The total space complexity is ,when The time was significantly reduced.

[0121] S3. Principal component extraction is performed on the influence function of the knowledge retention set through singular value decomposition to construct a knowledge subspace basis; the influence function of the privacy sample is projected onto the knowledge subspace basis and orthogonally decomposed to obtain knowledge components and privacy components; wherein, the knowledge component represents the part of the influence of the privacy sample on the model parameters that is consistent with the direction of the influence of the parameters of the knowledge retention set; the privacy component represents the part of the influence of the privacy sample on the model parameters that is orthogonal to the direction of the influence of the parameters of the knowledge retention set.

[0122] For privacy samples Its influence function It comprises two parts: a knowledge component, which shares the influence direction with the retained set and represents the encoding of general domain knowledge; and a privacy component, which shares the influence direction with privacy samples and represents the encoding of personal privacy information. Decomposition is achieved through the projection of the influence function onto the retained set. Step S3 specifically includes S31~S34.

[0123] S31. Obtain the set of influence functions for all samples in the knowledge retention set: .

[0124] S32. Extraction via singular value decomposition Main direction of influence: ;in Represents the singular value decomposition function; It is a left singular vector matrix, that is, orthogonal to the parameter space; It is a singular value diagonal matrix. ,satisfy ; It is a right singular vector matrix;

[0125] S33. From Take the middle before Principal components (i.e., before taking the first principal component) (Columns) constitute the basis of the knowledge subspace: .in The choice makes the cumulative variance contribution rate greater than .

[0126] S34. Regarding privacy samples Decompose its influence function into knowledge components and privacy :

[0127] ;

[0128] Regarding the amount of knowledge , representing the portion of the influence of privacy samples on parameters that overlaps with the reserved set, i.e., the encoding of general knowledge.

[0129] Regarding privacy, This indicates the unique impact of privacy samples, specifically the encoding of privacy data for particular users. It is an identity matrix.

[0130] S4. Design a parameter correction strategy based on the knowledge component and the privacy component, and update the model parameters according to the parameter correction strategy. While eliminating the privacy impact along the privacy component direction, compensate for knowledge loss along the knowledge component direction, thereby iteratively optimizing the large language model.

[0131] Step S4 specifically includes S41 to S44.

[0132] S41. Calculate the ratio of the magnitude of the privacy component to the magnitude of the knowledge component, and allocate each privacy sample according to the ratio. It is divided into three levels: high risk, medium risk, and low risk. Specifically:

[0133] like If the privacy sample is high-risk, privacy information dominates, and complete forgetting occurs.

[0134] like If the privacy sample is of medium risk, then privacy information and knowledge coexist, and selective forgetting occurs.

[0135] like If the privacy sample is low-risk, the knowledge redundancy is high, and there is slight forgetting.

[0136] S42. Calculate the privacy impact of each layer of the model using the following formula:

[0137] ;

[0138] In the formula, For the first A forgotten privacy sample In the model The impact of layered privacy; For the first Privacy components of a privacy sample In the Layer component vectors; aggregated hierarchical privacy impact The larger the value, the higher the value of the model. The stronger the encoding of privacy information in each layer; It is an L2 norm.

[0139] S43. Identify the privacy-critical layers based on the privacy impact of each layer, i.e., the layers that need to be prioritized and forgotten:

[0140] ;

[0141] In the formula, For privacy-critical layers; Privacy impact score after traversing all L layers of the model The maximum value is used as the benchmark threshold for determining the privacy-critical layer.

[0142] This invention utilizes the decomposed privacy impact components to guide parameter updates and designs differentiated parameter correction strategies to achieve the goal of "accurately forgetting privacy and retaining general knowledge".

[0143] The principle behind function-driven parameter correction is as follows:

[0144] Influence function Indicates removal of samples The direction in which the parameters should change. To forget privacy samples while preserving knowledge, parameter adjustments should follow the direction of the privacy component to eliminate privacy effects, while simultaneously offsetting the effects of the knowledge component to avoid knowledge loss. The parameter updates are as follows:

[0145] ;

[0146] in: The total number of privacy samples to be forgotten. For the first The forgetting weight for each sample can be determined based on the privacy risk level. and These are the privacy correction coefficient and the knowledge compensation coefficient, respectively, adjusted according to the risk level of the privacy samples. For high-risk samples, the sample weights are set... Privacy correction factor Intensity of forgetting; knowledge compensation coefficient Mild compensation is used to avoid excessive privacy protection. For medium-risk samples, the sample weights are adjusted. Privacy correction factor Knowledge compensation coefficient Balancing forgetting and retention. For low-risk samples, sample weights... Privacy correction factor Mild forgetting; knowledge compensation coefficient Prioritize the retention of knowledge.

[0147] Based on the above principles, this invention employs differentiated updates for different layers according to hierarchical privacy impact analysis, thus including step S44.

[0148] S44. Use a differentiated update strategy for the parameters of different layers of the model:

[0149] ;

[0150] In the formula, The total number of privacy samples to be forgotten; For the first Forgetting weights for privacy samples and These are the privacy correction coefficient and the knowledge compensation coefficient, all of which are determined based on the risk level of the privacy sample. For the model number The original optimal parameters of the layer; For the model after the forgetting operation New parameters for the layer; For the first The corrected strength of the layer; For the first The knowledge components of a privacy sample in the th Layer component vectors.

[0151] That is, for the key layer For non-critical layers, a full privacy information and knowledge separation correction is performed, while for non-critical layers, only minor knowledge enhancement is applied to avoid unnecessary disturbances. The intensity of the hierarchical correction is as follows. A hierarchical decreasing strategy is adopted:

[0152] ;

[0153] in , This ensures that the correction range in shallow layers is greater than that in deep layers.

[0154] In addition, to prevent parameter corrections from deviating excessively from the original model, a global parameter offset constraint is introduced.

[0155] Define the regularization loss for parameter offsets. The constraint is: the parameter correction should satisfy... ;in This means that the overall parameter variation is allowed to be no more than 1%, to avoid excessive parameter correction that could cause the model performance to collapse.

[0156] If the parameter adjustment exceeds the constraint, adaptive scaling is performed, as shown in the following expression:

[0157] ;

[0158] In the formula, This is the scaling factor. ; To obtain the minimum value. If ,but No scaling is required; if it exceeds the constraints, then... The correction range is scaled up to just meet the constraints.

[0159] This invention can employ different update schemes depending on the number of forgotten samples.

[0160] In step S4, if the total number of privacy samples to be forgotten... If the number of samples is small-scale forgetting, it will be updated using the parameter correction formula mentioned above. As for the total number of privacy samples to be forgotten... Then, multiple rounds of iterative optimization and updates are adopted, and the impact function is re-evaluated after each correction.

[0161] First, score by influence. Sort the privacy samples in descending order and divide the samples into Each batch consists of a maximum of 50 samples, and samples are forgotten sequentially according to batch order. The model parameters and influence function subspace projection matrix of the previous batch are reused in each batch. The influence function is calculated in parallel for samples within a batch, and the parameters are updated serially between batches, balancing efficiency and effectiveness.

[0162] The process employs multiple rounds of iterative convergence, with each round consisting of B batches of sequential processing, as detailed below:

[0163] Step 1: Initialization

[0164] Initialize the model parameters to the baseline parameters, that is: , This represents the initial parameters for round 0 and batch 0, providing a starting point for subsequent iterations.

[0165] Step 2: Multiple rounds and batches of iteration. For each round of iteration... t ( t =1,2, , T ), and each batch of samples within that round. ( ), perform the following operations:

[0166] Sub-step 2.1: Calculate the sample influence function based on the current parameters (No. t -Parameters after the first iteration), for batch b Each sample in Calculate their privacy-related impact functions respectively. And the influence function of knowledge retention This study quantifies the specific impact of each sample on the two objectives of "privacy protection" and "knowledge preservation," providing a basis for parameter correction.

[0167] Sub-step 2.2: Parameter Correction. Based on the sample influence function and corresponding coefficients, the parameters of the current batch are corrected. The calculation formula is as follows:

[0168] ;

[0169] in, For the first t Wheel, First b -1 Parameters after batch processing. The goal is to adjust the parameters in a direction that "enhances privacy protection and minimizes knowledge loss" by weighted summation and fusion of the privacy and knowledge impacts of the samples.

[0170] Sub-step 2.3: Calculate the current parameters for regularization constraints. With reference parameters The offset; if the offset exceeds the threshold Then for Perform scaling to ensure: This is to avoid the parameters deviating from the baseline model due to excessive correction, and to ensure the stability of the model's core performance.

[0171] Sub-step 2.4: After correcting the parameters for the current batch, dynamically adjust the coefficients:

[0172] ;

[0173] As the iteration progresses, the intensity of privacy correction is gradually reduced to finely adjust the parameters, and the knowledge retention coefficient is enhanced to reduce the loss of the model's original knowledge.

[0174] Step 3: Iteration terminates upon completion T After round iteration and all batch processing, or when the early stopping condition is met, the current parameter will be... As the optimized model parameters.

[0175] Training can be terminated if any of the following conditions are met during the iterative correction process:

[0176] Condition 1: Adequacy of forgetting is met

[0177] Define a privacy loss function and evaluate the forgetting effect: This function measures the model's average loss on the samples to be forgotten. A larger value indicates a worse fit to the forgotten samples; a smaller value indicates that the model still remembers the forgotten samples. The iteration terminates at a certain point, where the privacy loss threshold is set. Based on the forgetting target: Strong forgetting scenarios can be set. Standard forgetting scenarios can be set. For mild forgetfulness scenarios, the following settings can be configured: .

[0178] Condition 2: Reaching the maximum number of iterations

[0179] when The iteration terminates at that time. The maximum number of iteration rounds is set to 20 in this embodiment.

[0180] Condition 3: Loss convergence

[0181] when If the condition is met for 5 consecutive rounds, the iteration stops.

[0182] After iteration, a large language model with privacy forgotten is obtained, which can be applied to relevant vertical question-answering systems, such as medical and health consultation, financial risk control question-answering, legal knowledge consultation and other vertical field applications; the specific steps are: input the question text into the large language model with privacy forgotten, and output the response text.

[0183] This embodiment uses the batch forgetting of customer privacy data in a large-scale financial model as a specific implementation example. A certain financial language model is based on the Llama2-70B architecture, containing an 80-layer Transformer structure with 64 attention heads per layer. It has been continuously pre-trained on general financial corpora, market analysis reports, and customer service dialogue data, achieving a high level of financial consultation and risk assessment capabilities. However, after deployment, it was found that due to incomplete anonymization of the training data, the model remembered some real customer privacy information. These records include customers' real identity information, specific transaction details, asset allocation details, investment preferences, and other private content. According to relevant laws and regulations and user deletion requests, it is necessary to completely forget these customer privacy records while preserving the model's financial expertise.

[0184] Step 1: Data Preparation and Preprocessing. Collect all training samples related to this batch of customers. First, construct a forgotten dataset, covering customer consultation dialogue records, transaction record summaries, investment advice texts, etc., obtaining approximately 2000 training samples with privacy leakage risks. This data contains identifiable personal privacy information, such as customer's real name, ID number, bank account number, transaction amount, asset allocation details, etc., which need to be completely forgotten from the model. Simultaneously, construct a knowledge retention reference set, collecting three types of samples: The first type consists of samples of similar financial Q&A that do not involve specific customer information. Identifiers such as customer name, account number, and specific transaction amount are replaced with generic placeholders, transforming personalized investment portfolios into abstract descriptions of asset allocation principles. For example, the question "How should 500,000 yuan in Zhang's account be allocated?" could be rewritten as "What are the general allocation principles for 500,000 yuan of funds for a conservative investor?" The second category consists of general financial knowledge samples, including descriptions of the characteristics and investment points of various financial products, modern portfolio theory, and publicly available knowledge about market mechanisms, trading rules, and financial terminology from financial encyclopedias. The third category comprises generalized samples from other financial fields, used to assess and maintain the model's general capabilities in various sub-fields such as stock analysis, bond investment, insurance planning, foreign exchange trading, futures and options, and wealth management, ensuring that the forgetting process does not impair the model's overall financial expertise. While the retained reference set and the forgotten dataset overlap in terms of financial expertise, they are orthogonal to each other in terms of specific client privacy information.

[0185] Step 2: Calculate the influence function for samples in the forgotten dataset and the retained reference set respectively. Since the total number of model parameters reaches hundreds of billions, directly calculating the complete Hessian matrix is ​​infeasible. Therefore, a low-rank subspace projection technique is used to achieve efficient approximation, projecting the original high-dimensional parameter space into a subspace with significantly reduced dimensionality. Within this low-dimensional subspace, using the Hessian vector product of the automatic differentiation framework, a dimensionality-reduced Hessian approximation matrix is ​​gradually constructed by performing mini-batch sampling and gradient calculation on the retained reference set. After solving for the inverse matrix of this dimensionality-reduced matrix, the complete influence vector of the sample set on the parameters of each layer of the model is obtained.

[0186] Step 3: Privacy Knowledge Decoupling and Risk Classification. This step decomposes the parameter impact of privacy samples into privacy components and knowledge components, and classifies the risks accordingly. First, a knowledge retention subspace is extracted. The impact vectors of all samples in the retained reference set are organized into an impact matrix, and singular value decomposition is performed on it. Several principal components are selected based on the cumulative variance contribution rate to form an orthogonal basis. This subspace characterizes the main influence directions of general financial knowledge in the model parameter space. Then, orthogonal decomposition is performed on each privacy sample in the forgotten dataset. The knowledge component is obtained by projecting the impact vector onto the knowledge retention subspace, representing the parameter impact shared by the sample with general financial knowledge; the privacy component is obtained by calculating the projection on the orthogonal complement space, representing the parameter impact unique to the sample and related to specific customer privacy information. Analysis shows that samples containing real customer identity information and specific financial details have significantly higher privacy specificity, while samples mainly involving financial theory references and general investment strategies have lower privacy specificity.

[0187] Based on the ratio of privacy components to knowledge components, the samples are divided into three risk levels: high-risk samples contain strong privacy identifiers such as the customer's real identity, bank account number, securities account, and specific transaction amount; medium-risk samples contain mixed information such as personalized investment advice and specific account analysis; and low-risk samples mainly consist of personalized expressions of general financial knowledge. Simultaneously, a privacy impact analysis is performed on each layer of the model, and layers with a cumulative privacy component strength exceeding a certain proportion of the highest layer are marked as privacy-critical layers, serving as the focus of parameter correction.

[0188] Step 4: Parameter Correction and Iterative Optimization. Based on the decoupling results, a differentiated parameter correction strategy is designed and iterative optimization is performed. At the sample level, a higher forgetting intensity coefficient and a lower knowledge compensation coefficient are used for high-risk samples to ensure sufficient elimination of privacy information; for low-risk samples, a lower forgetting intensity and a higher knowledge compensation coefficient are used to prioritize the retention of financial knowledge components. At the hierarchical level, a complete bidirectional parameter correction is performed on the privacy-critical layer, i.e., strengthening forgetting along the privacy component direction and actively compensating along the knowledge component direction; only a slight knowledge enhancement operation is performed on non-critical layers. The hierarchical correction intensity adopts a strategy of decreasing from shallow to deep layers.

[0189] Due to the large number of forgotten samples, a multi-round, multi-batch iterative optimization scheme was adopted. Samples were sorted in descending order of privacy component strength and processed in batches. After each batch, the influence function was re-evaluated and the correction coefficients were dynamically adjusted. Global constraints on parameter offsets were introduced to prevent over-correction. Three termination conditions were set: when the model's average loss on forgotten samples exceeded a preset threshold, it indicated that forgetting was sufficient and the model could terminate; when the number of iterations reached a preset upper limit, termination was forced to avoid overcomputation; when the loss change was less than a convergence threshold for several consecutive iterations, it indicated that the optimization had stabilized and termination was initiated early. The final output is a model that fully forgets customer privacy information while maintaining financial expertise.

[0190] To verify the effectiveness of the method of this invention, it was compared with mainstream baseline methods in three dimensions: completeness of forgetting, knowledge retention, and computational efficiency. The comparison was conducted on a server equipped with eight NVIDIA A800 80GB GPUs, and the following four representative baseline methods were selected for comparison:

[0191] Fine-tuning: Fine-tuning is performed on a randomly labeled version of the forgotten sample, which disrupts the model's memory of private information by introducing noisy labels.

[0192] Gradient Ascent: This method performs gradient ascent on forgotten samples, directly increasing the model's loss on these samples. It is the most direct forgetting method.

[0193] ROME: Based on causal tracing, it locates the knowledge storage location and modifies specific fact memories through rank-one editing, suitable for precise editing of single knowledge items.

[0194] MEMIT: A batch extension of ROME, supporting the simultaneous editing of multiple knowledge entries and achieving batch forgetting through distributed modification of multi-layer MLP parameters.

[0195] For the forgetting effect metrics: Forgetting Rate (FR) is the proportion of privacy information in forgotten samples that the model cannot correctly generate; Privacy Leakage Risk (PLR) is the probability that the model leaks privacy information under various probing attacks. For the knowledge retention metrics: Financial Knowledge Accuracy (FKA) is the model's question-answering accuracy on the financial knowledge test set; Knowledge Retention Rate (KRR) is the ratio of the model's knowledge capability after forgetting to the combined knowledge capability of the original model. For the efficiency metrics: Computation Time (CT) is the total time required to complete the forgetting process; Parameter Correction Ratio (PMR) is the proportion of corrected parameters to the total number of parameters. The results are shown in Table 1.

[0196] Table 1: Comparison of Forgetting Effects and Knowledge Retention

[0197]

[0198] Experimental results show that the method of this invention outperforms the baseline method in terms of forgetting performance, achieving a forgetting rate of 80.47% and reducing the risk of privacy leakage to 33.26%. In terms of knowledge retention, the knowledge retention rate reaches 82.34%.

[0199] Example 2

[0200] This embodiment provides a large-scale model privacy knowledge decoupling and forgetting system based on influence function decomposition, applying the large-scale model privacy knowledge decoupling and forgetting method based on influence function decomposition as described above; the system includes:

[0201] The influence function calculation module is used to obtain the privacy sample set to be forgotten and the knowledge retention set for maintaining general capabilities; construct a random orthogonal projection matrix to project the Hessian matrix of the large language model parameter space to a low-dimensional subspace; and calculate the influence function of the privacy sample set and the samples in the knowledge retention set based on the low-dimensional subspace to quantify the impact of removing samples on the model parameters.

[0202] The privacy knowledge decomposition module is used to extract principal components from the influence function of the knowledge retention set and construct a knowledge subspace basis; the influence function of the privacy sample is projected onto the knowledge subspace basis to decompose it into knowledge components and privacy components.

[0203] A selective forgetting execution module is used to design a parameter correction strategy based on the knowledge component and the privacy component;

[0204] The model parameter update module is used to design parameter correction strategies based on the knowledge components and the privacy components, and update the model parameters according to the parameter correction strategies. While eliminating the privacy impact along the privacy component direction, it compensates for knowledge loss along the knowledge component direction, thereby iteratively optimizing the large language model.

[0205] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A privacy knowledge decoupling and forgetting method based on large-scale model influence function decomposition, characterized in that, include: S1. Obtain the set of privacy samples to be forgotten and the knowledge retention set for maintaining general capabilities; For knowledge-preserving samples, a generic placeholder is used to replace the text. S2. Construct a random orthogonal projection matrix, and project the Hessian matrix of the large language model parameter space to a low-dimensional subspace through the projection matrix. Solve the Hessian inverse matrix in the low-dimensional subspace, and calculate the influence function of the privacy sample set and the samples in the knowledge retention set based on the inverse matrix, so as to quantify the impact of removing samples on the model parameters. S3. Principal component extraction is performed on the influence function of the knowledge retention set through singular value decomposition to construct a knowledge subspace basis; the influence function of the privacy sample is projected onto the knowledge subspace basis and orthogonally decomposed to obtain knowledge components and privacy components; wherein, the knowledge component represents the part of the influence of the privacy sample on the model parameters that is consistent with the direction of the influence of the parameters of the knowledge retention set; the privacy component represents the part of the influence of the privacy sample on the model parameters that is orthogonal to the direction of the influence of the parameters of the knowledge retention set. S4. Design a parameter correction strategy based on the knowledge component and the privacy component, and update the model parameters according to the parameter correction strategy. While eliminating the privacy impact along the privacy component direction, compensate for knowledge loss along the knowledge component direction, thereby iteratively optimizing the large language model.

2. The method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition according to claim 1, characterized in that, Step S2 specifically includes: S21. Generate a random orthogonal projection matrix : In the formula, , This represents the total number of model parameters. Let be the subspace dimension, and Less than , Let be the set of real numbers, satisfying superscript It is the transpose symbol. for An identity matrix of order 1; This indicates that after performing QR decomposition on the matrix, only the orthogonal matrix Q is taken; This indicates a range from a mean of 0 and a standard deviation of 0. Sampling in Gaussian distribution The elements constitute the initial matrix; S22. Divide the training set used to train the large language model into... The batch number, recorded as the [number]th batch The average Hessian matrix of each batch of samples is: ;in ; S23. Project the matrix Decomposed into columns Calculate the Hessian-vector product for each column separately. spliced ​​together Then calculate each batch Projection results Finally, for The low-dimensional subspace is obtained by summing and averaging the projection results of each batch. ;in ; S24. Solving for the regularized inverse matrix in low-dimensional space , ;in The damping coefficient; S25. For the target samples in the privacy sample set or the knowledge retention set. The influence function is calculated using the following formula: In the formula, For the sample Influence function; For target samples In optimal parameters The loss gradient at that point.

3. The method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition according to claim 2, characterized in that, In step S2, the computational efficiency of the influence function is improved in the following ways: Based on model characteristics, it is divided into Each layer calculates its influence function independently; the hierarchical influence function decomposition is expressed as: In the formula, the symbol This indicates vector concatenation; For layer index; For target samples In the model The influence function of the layer is calculated using the following formula: In the formula, For the model number Layer parameters, This is the projection matrix of this layer; For target samples In the optimal parameters of the model The loss function at point 1 for the first point 2 Layer parameters The gradient vector; In the case of batch forgetting multiple samples, the first sample is executed completely. , and The calculation is performed so that subsequent samples only calculate the new gradient and reuse the previously calculated gradient. and .

4. The method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition according to claim 3, characterized in that, Step S3 specifically includes: S31. Obtain the set of influence functions for all samples in the knowledge retention set: ;in For knowledge reservation; S32. Extraction via singular value decomposition Main direction of influence: ;in Represents the singular value decomposition function; It is a left singular vector matrix, that is, orthogonal to the parameter space; It is a singular value diagonal matrix; It is a right singular vector matrix; S33. From Take the front Each principal component constitutes a knowledge subspace basis: ; S34. Regarding privacy samples Decompose its influence function into knowledge components. and privacy ;in, , ; It is an identity matrix.

5. The method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition according to claim 4, characterized in that, Step S4 specifically includes: S41. Calculate the ratio of the magnitude of the privacy component to the magnitude of the knowledge component, and allocate each privacy sample according to the ratio. It is divided into three levels: high risk, medium risk, and low risk. S42. Calculate the privacy impact of each layer of the model using the following formula: In the formula, For the first A forgotten privacy sample In the model The impact of layered privacy; For the first Privacy components of a privacy sample In the Layer component vectors; aggregated hierarchical privacy impact The larger the value, the higher the value of the model. The stronger the encoding of privacy information in each layer; It is an L2 norm; S43. Identify the privacy-critical layers based on the privacy impact of each layer, i.e., the layers that need to be prioritized and forgotten: In the formula, For privacy-critical layers; Privacy impact score after traversing all L layers of the model The maximum value is used as the benchmark threshold for determining the privacy-critical layer; S44. Use a differentiated update strategy for the parameters of different layers of the model: In the formula, The total number of privacy samples to be forgotten; For the first Forgetting weights for privacy samples and These are the privacy correction coefficient and the knowledge compensation coefficient, all of which are determined based on the risk level of the privacy sample. For the model number The original optimal parameters of the layer; For the model after the forgetting operation New parameters for the layer; For the first The corrected strength of the layer; For the first The knowledge components of a privacy sample in the th Layer component vectors.

6. The method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition according to claim 5, characterized in that, In step S41, if If the privacy sample is high-risk, then the sample forgetting weight is applied. Privacy correction factor Knowledge compensation coefficient ; like If the privacy sample is considered medium risk, then the sample forgetting weight is... Privacy correction factor Knowledge compensation coefficient ; like If the privacy sample is low risk, then the sample forgetting weight is applied. Privacy correction factor Knowledge compensation coefficient .

7. The method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition according to claim 5, characterized in that, In step S44, a global parameter offset constraint is also introduced: Define the regularization loss for parameter offsets. The constraint is: the parameter correction should satisfy... ;in This means that the overall variation in parameters is allowed to be no more than 1%; If the parameter adjustment exceeds the constraint, adaptive scaling is performed, as shown in the following expression: In the formula, This is the scaling factor. ; To obtain the minimum value.

8. The method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition according to claim 5, characterized in that, In step S4, if the total number of privacy samples to be forgotten... If the threshold is exceeded, multiple rounds of iterative optimization and updates will be used: The privacy samples are divided into multiple batches and forgotten sequentially according to batch order. In each batch of each iteration, the privacy and knowledge components of the sample are calculated based on the current parameters, and parameter corrections and regularization constraints are applied. The privacy correction coefficient is dynamically decayed as the iteration progresses. And increase the knowledge compensation coefficient The iteration terminates when the privacy loss threshold, maximum number of iterations, or loss convergence condition is met.

9. The method for decoupling and forgetting privacy knowledge in large models based on influence function decomposition according to claim 1, characterized in that, In step S1, the knowledge retention set contains three types of samples: the first type is non-privacy samples of the same task, which belong to the same task type as privacy samples but do not contain sensitive information; The second category is general domain knowledge samples, which contain encyclopedic knowledge or professional guides in relevant fields; the third category is domain generalization samples, which are used to evaluate the general reasoning ability of the model; wherein, the knowledge retention set is constructed to overlap with the privacy sample set in terms of domain knowledge features, but to be orthogonal to the privacy sample set in terms of privacy information features.

10. A large-scale privacy knowledge decoupled forgetting system based on influence function decomposition, characterized in that, The system employs the large-model privacy knowledge decoupling and forgetting method based on influence function decomposition as described in any one of claims 1 to 9; the system comprises: The influence function calculation module is used to obtain the privacy sample set to be forgotten and the knowledge retention set for maintaining general capabilities; construct a random orthogonal projection matrix, and project the Hessian matrix of the large language model parameter space to a low-dimensional subspace through the projection matrix. Solve the Hessian inverse matrix in the low-dimensional subspace, and calculate the influence function of the privacy sample set and the samples in the knowledge retention set based on the inverse matrix to quantify the impact of removing samples on the model parameters; The privacy knowledge decomposition module is used to extract principal components from the influence function of the knowledge retention set through singular value decomposition to construct a knowledge subspace basis; the influence function of the privacy sample is projected onto the knowledge subspace basis and orthogonally decomposed to obtain knowledge components and privacy components; wherein, the knowledge component represents the part of the influence of the privacy sample on the model parameters that is consistent with the influence direction of the parameters of the knowledge retention set; the privacy component represents the part of the influence of the privacy sample on the model parameters that is orthogonal to the influence direction of the parameters of the knowledge retention set. A selective forgetting execution module is used to design a parameter correction strategy based on the knowledge component and the privacy component; The model parameter update module is used to design parameter correction strategies based on the knowledge components and the privacy components, and update the model parameters according to the parameter correction strategies. While eliminating the privacy impact along the privacy component direction, it compensates for knowledge loss along the knowledge component direction, thereby iteratively optimizing the large language model.

Images