A text privacy protection method and system based on subspace projection large language model forgetting
By introducing linear projection and recalibration layers into the Transformer layer of the large language model, and utilizing subspace projection technology to purify privacy information, the problem of separating privacy text from general text representation is solved, achieving a balance between efficient privacy protection and model performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HARBIN INST OF TECH
- Filing Date
- 2026-02-25
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to effectively separate the representations of private text from those of general public text, making it difficult to protect the privacy information within the text.
A large language model forgetting method based on subspace projection is adopted. By introducing a linear projection and recalibration layer in the Transformer layer, a forgetting subspace projection matrix is constructed using the forgetting set and the retention set. The original hidden state is then projected and recalibrated to achieve the purification and stripping of privacy information.
It achieves efficient protection of text privacy information, ensuring that the model maintains good logical expression ability and knowledge richness while forgetting privacy information, and reduces computational overhead and deployment costs.
Smart Images
Figure CN122133183A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, specifically to a method and system for protecting text privacy based on the forgetting of a large language model with subspace projection. Background Technology
[0002] Current methods for protecting privacy and handling forgotten information in text primarily rely on techniques such as retraining large language models and fine-tuning based on gradient backpropagation. For example, gradient ascent algorithms are used to forcibly increase the model's loss function for target texts containing private or copyrighted content; negative preference optimization and direct preference optimization methods are employed to reduce the generation probability of texts containing sensitive information; or techniques such as neural activation redirection are used to intervene in the model's internal representation of the target private text, weakening its retention and output capabilities. However, because the representations of private texts, copyrighted texts, and general public text information are highly coupled and difficult to separate, existing technologies struggle to protect the privacy information within text. Summary of the Invention
[0003] The purpose of this invention is to address the problem that existing technologies struggle to protect privacy information in texts due to the high coupling and difficulty in separating the representations of private texts, copyrighted texts, and general public text information. This invention provides a text privacy protection method and system based on the forgetting mechanism of a large language model using subspace projection.
[0004] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:
[0005] A text privacy protection method based on subspace projection large language model forgetting includes the following steps:
[0006] For the text of the literary work to be protected, the text of the literary work to be protected is input into the large language model, and the output privacy-protected text is obtained.
[0007] The large language model has at least one Transformer layer followed by a linear projection and recalibration layer, which utilizes a forgetting subspace projection matrix obtained based on the forgetting set and the retained set. For the original hidden state Perform linear projection and recalibration to obtain the hidden state. , is represented as:
[0008] ;
[0009] in, This is the hidden state after the transformation. It is in its original hidden state.
[0010] Furthermore, the forgotten subspace projection matrix It is obtained through the following steps:
[0011] Step 1: Obtain the text of the written work, mark the privacy information in the text, and construct a dataset accordingly;
[0012] Step 2: Divide the dataset into a forget set Sf and a retention set Sr;
[0013] Step 3: For each data point in the forget set Sf, input the data into the basic large language model and obtain the hidden state corresponding to each token in the Transformer layer. ,in, Indicates the sample index. This represents the token index. Then, average pooling or weighted pooling is applied to all hidden states within each Transformer layer to obtain the sample-level activation vector, represented as:
[0014] ;
[0015] Step 4: Repeat step 3 to obtain the sample-level activation vector corresponding to each data point in the forgetting set Sf, and use the sample-level activation vectors corresponding to all data points to obtain the representation matrix. ;
[0016] Step 5: For the retained set Sr, repeat steps 3 and 4 to obtain the representation matrix corresponding to the retained set Sr. ;
[0017] Step 6: For the representation matrix After centralization, singular value decomposition is performed, which is represented as:
[0018] ;
[0019] in, The left-hand unitary matrix after decomposition. The resulting diagonal matrix is the decomposed matrix. The decomposed right-hand unitary matrix, Let be the mean vector of the matrix;
[0020] Step 7: According to Construct a base for preserving subspaces ;
[0021] Step 8: Utilize the preserved subspace basis The projection matrix of the preserved subspace is obtained. , is represented as:
[0022] ;
[0023] Step 9: Based on the representation matrix and the reserved subspace projection matrix The purified representation matrix is obtained. , is represented as:
[0024] ;
[0025] in, It is the identity matrix;
[0026] Step 10: Process the purified representation matrix Perform singular value decomposition, represented as:
[0027] ;
[0028] in, The left-hand unitary matrix after decomposition. The resulting diagonal matrix is the decomposed matrix. The right-hand unitary matrix after decomposition
[0029] Step 11: According to Constructing the forgetting subspace basis ;
[0030] Step 12: Utilize the forgotten subspace basis The projection matrix of the forgotten subspace is obtained. , is represented as:
[0031] .
[0032] Furthermore, the specific steps of step 8 are as follows:
[0033] exist In the middle, before selecting by singular value energy In one direction, construct a preserved subspace basis. ,in,
[0034] ;
[0035] in, For the first matrix A singular value, It is a constant. Let be the number of singular values of the matrix. For the first matrix One singular value;
[0036] The specific steps of step 12 are as follows:
[0037] exist In the middle, before selecting by singular value energy One direction for constructing the forgetting subspace basis. ,in,
[0038] ;
[0039] in, For the first matrix A singular value, It is a constant.
[0040] Furthermore, in step 1, the privacy information in the text is marked through semantic retrieval or manual annotation.
[0041] Furthermore, the underlying model is Llama3-3b-Instruct or LlaMa3-8b-Instruct.
[0042] A text privacy protection system based on subspace projection large language model forgetting, the system specifically performs the following steps:
[0043] For the text of the literary work to be protected, the text of the literary work to be protected is input into the large language model, and the output privacy-protected text is obtained.
[0044] The large language model has at least one Transformer layer followed by a linear projection and recalibration layer, which utilizes a forgetting subspace projection matrix obtained based on the forgetting set and the retained set. For the original hidden state Perform linear projection and recalibration to obtain the hidden state. , is represented as:
[0045] ;
[0046] in, This is the hidden state after the transformation. It is in its original hidden state.
[0047] Furthermore, the forgotten subspace projection matrix It is obtained through the following steps:
[0048] Step 1: Obtain the text of the written work, mark the privacy information in the text, and construct a dataset accordingly;
[0049] Step 2: Divide the dataset into a forget set Sf and a retention set Sr;
[0050] Step 3: For each data point in the forget set Sf, input the data into the basic large language model and obtain the hidden state corresponding to each token in the Transformer layer. ,in, Indicates the sample index. This represents the token index. Then, average pooling or weighted pooling is applied to all hidden states within each Transformer layer to obtain the sample-level activation vector, represented as:
[0051] ;
[0052] Step 4: Repeat step 3 to obtain the sample-level activation vector corresponding to each data point in the forgetting set Sf, and use the sample-level activation vectors corresponding to all data points to obtain the representation matrix. ;
[0053] Step 5: For the retained set Sr, repeat steps 3 and 4 to obtain the representation matrix corresponding to the retained set Sr. ;
[0054] Step 6: For the representation matrix After centralization, singular value decomposition is performed, which is represented as:
[0055] ;
[0056] in, The left-hand unitary matrix after decomposition. The resulting diagonal matrix is the decomposed matrix. The decomposed right-hand unitary matrix, Let be the mean vector of the matrix;
[0057] Step 7: According to Construct a base for preserving subspaces ;
[0058] Step 8: Utilize the preserved subspace basis The projection matrix of the preserved subspace is obtained. , is represented as:
[0059] ;
[0060] Step 9: Based on the representation matrix and the reserved subspace projection matrix The purified representation matrix is obtained. , is represented as:
[0061] ;
[0062] in, It is the identity matrix;
[0063] Step 10: Process the purified representation matrix Perform singular value decomposition, represented as:
[0064] ;
[0065] in, The left-hand unitary matrix after decomposition. The resulting diagonal matrix is the decomposed matrix. The right-hand unitary matrix after decomposition
[0066] Step 11: According to Constructing the forgetting subspace basis ;
[0067] Step 12: Utilize the forgotten subspace basis The projection matrix of the forgotten subspace is obtained. , represented as:
[0068] .
[0069] Furthermore, the specific steps of step 8 are as follows:
[0070] exist In the middle, before selecting by singular value energy In one direction, construct a preserved subspace basis. ,in,
[0071] ;
[0072] in, For the first matrix A singular value, It is a constant. Let be the number of singular values of the matrix. For the first matrix One singular value;
[0073] The specific steps of step 12 are as follows:
[0074] exist In the middle, before selecting by singular value energy One direction for constructing the forgetting subspace basis. ,in,
[0075] ;
[0076] in, For the first matrix A singular value, It is a constant.
[0077] Furthermore, in step 1, the privacy information in the text is marked through semantic retrieval or manual annotation.
[0078] Furthermore, the underlying model is Llama3-3b-Instruct or LlaMa3-8b-Instruct.
[0079] The beneficial effects of this invention are:
[0080] This application achieves efficient purification and stripping of specific textual privacy information by implementing precise orthogonal decoupling and projection intervention in the high-dimensional feature layer inside the model, which can effectively protect the privacy information in the text. Attached Figure Description
[0081] Figure 1 This is a framework diagram of the large language model based on subspace projection in this application;
[0082] Figure 2 This is a comparison chart of time taken on the TOFU dataset. Detailed Implementation
[0083] It should be noted that, where there is no conflict, the various embodiments disclosed in this application can be combined with each other.
[0084] Specific Implementation Method 1: The text privacy protection method based on subspace projection large language model forgetting described in this implementation method includes the following steps:
[0085] Step 1: Define Data Segmentation and Task Objectives. For the textual works to be protected, define a privacy dataset using semantic retrieval or manual annotation. Divide the original training data into a forget set Sf and a retention set Sr. The forget set contains sensitive or restricted text samples that need to be removed from the model; the retention set contains other training data and general knowledge that need to be retained.
[0086] Step 2: Select the base model and layers. Determine the base model to be processed (e.g., LLaMA-3, LLaMA-3-Instruct, etc.) and select the set of network layers L for intervention (which can be all Transformer layers or a select few layers).
[0087] Step 3: Sample Input and Hidden State Acquisition. For each sample x in Sf and Sr, input each sample into the model, and then acquire the hidden state of that sample for each token in each layer l. ,in Indicates the sample index. (This refers to the token index).
[0088] Step 4: Sample-level representation aggregation. Average pooling or weighted pooling is applied to all hidden states within each layer to obtain sample-level activation vectors. To form a matrix and , represent the representation matrices of the forgotten set and the retained set in layer l, respectively.
[0089] Step 5: Center the retained representation. Calculate the mean of the retained set representation and center it. Calculate the mean vector of the retained set representation matrix and subtract this mean from each sample.
[0090] Step 6: Perform Singular Value Decomposition (SVD) on the representation matrix of the retained set. ;
[0091] from Before selecting by singular value energy Each direction constitutes a basis for the preserved subspace. The left side of this SVD decomposition is the centered matrix, and the right side is the three matrices obtained from the decomposition.
[0092] ;
[0093] in, Let i be the i-th singular value of the matrix (sorted in descending order). It is a constant value given based on experience.
[0094] Step 7: Construct the projection matrix of the preserved subspace. Calculate the projection matrix of the preserved subspace. .
[0095] Step 8: Center the forgetting representation using the mean-preserving method. Center the forgetting set representation using the mean-preserving method to ensure coordinate system consistency.
[0096] Step 9: Project onto the orthogonal complement of the preserved subspace to cleanse the forgotten information. Calculate the cleaned-up forgotten representation. This operation removes the orientation shared by the subspaces that are preserved, where I is the identity matrix.
[0097] Step 10: Perform SVD again on the purified forgetting representation. Execute SVD: Choose the first one from the options. The main directions constitute the basis of the forgotten subspace. .
[0098] Step 11: Construct the forgetting subspace projection matrix. After constructing the forgetting subspace, the corresponding projection matrix is first calculated to extract the forgetting subspace components from any hidden state. Calculation , is used to project any vector onto the forgotten subspace.
[0099] Step 12: Subsequently, to avoid a significant decrease in the norm of the hidden state due to direct projection, we introduce a norm-preserving recalibration strategy to normalize and scale the representation after removing the forgotten components. Given the original hidden state, compute... .
[0100] Step 13: During the inference phase, the linear projection and recalibration operations are embedded into the model's forward propagation process to dynamically transform the hidden states of the specified layer. Through this mechanism, the model can continuously suppress representations related to the target privacy during forward computation without requiring parameter updates, thereby achieving effective text privacy anti-learning.
[0101] Step 14: The transformed model automatically performs anti-learning processing to protect the privacy of the target text during forward propagation, ultimately forming a compliant large language model with privacy protection capabilities. Through this process, this method achieves the protection of text privacy.
[0102] Additionally, the matrix Pf was obtained in 12 steps. Training is complete. In actual use, for the text input to the model, the hidden state hu of each layer is obtained sequentially, and each hidden state is transformed by projection, i.e., guided in the direction of forgetting.
[0103] Table 1. Experimental results on the TOFU dataset.
[0104]
[0105] Table 2 Experimental results on the Harry Potter dataset
[0106]
[0107] The main focus of this application is:
[0108] (a) indicates the extraction module: the model forwards and collects the hidden states of tokens in each layer, and performs sample-level aggregation to form Hf and Hr.
[0109] (b) Subspace construction: After centering Hr, perform SVD, select the first kr direction to form Br, and calculate the projection matrix Pr.
[0110] (c) Construction of the forgotten subspace: Project Hf onto (I−Pr) to purify it and then perform SVD on it, selecting the first kf direction to form Bf.
[0111] (d) Forgetting Projection Module: Calculate Pf and apply a recalibrated projection to the hidden state ho during inference to obtain hu, which is used to replace the original hidden state to complete the forgetting.
[0112] As can be seen from the above, the technical solution of this application has achieved the following technical effects:
[0113] Extremely high precision and thoroughness in privacy erasure: Figure 2The experimental data in Tables 1 and 2 show that the Forget Quality (FQ) of this application can reach 0.999 when processing the privacy forgetting task of written works. This means that the model can completely eliminate the memory of specific privacy fragments, sensitive personal information or copyrighted text, effectively solve the privacy residue problem existing in traditional methods, and meet the stringent legal compliance and right to be forgotten technical requirements.
[0114] Exceptional Preservation of Model Generality: Unlike traditional fine-tuning methods such as gradient ascent (GA) or partial sum optimization (NPO), this application utilizes subspace projection technology to achieve deep decoupling between privacy features and general semantic logic. While ensuring the interception of privacy information, the model utility (MU) remains consistently at a high level of around 0.63. This ensures that the model's logical expression ability and knowledge richness are not compromised when performing normal business tasks such as text creation and common sense question answering, effectively overcoming the serious drawbacks of catastrophic forgetting and performance coupling in existing technologies.
[0115] Extremely low computational overhead and high response speed: Because this application employs an inference-end intervention strategy based on linear algebraic transformations, it eliminates the need for expensive model parameter retraining or full fine-tuning. In terms of time efficiency, it achieves an improvement of over 80% compared to traditional fine-tuning-based forgetting methods. This technical characteristic enables the rapid deployment of this solution on large-scale pre-trained models, significantly reducing the computational and time costs for enterprises in maintaining privacy compliance.
[0116] Excellent adaptability and engineering usability: The subspace purification mechanism proposed in this application does not depend on the architecture of a specific model and has good versatility. Compared with methods such as LUNAR that rely on complex neuron redirection, the process of this application is simpler and more robust, and can be directly embedded into existing generative AI service pipelines, providing a highly reliable and scalable technical solution for real-world scenarios such as copyright protection of literary works and desensitization of trade secrets.
[0117] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solution of the present invention and should not be used to limit the scope of protection. Any modifications made in accordance with the claims and specification of the present invention that are only partial should still fall within the protection scope of the present invention.
Claims
1. A text privacy protection method based on subspace projection large language model forgetting, characterized in that... Includes the following steps: For the text of the literary work to be protected, the text of the literary work to be protected is input into the large language model to obtain the output privacy-protected text; The large language model has at least one Transformer layer followed by a linear projection and recalibration layer, which utilizes a forgetting subspace projection matrix obtained based on the forgetting set and the retained set. For the original hidden state Perform linear projection and recalibration to obtain the hidden state. , is represented as: ； in, This is the hidden state after the transformation. It is in its original hidden state.
2. The text privacy protection method based on subspace projection large language model forgetting as described in claim 1, characterized in that... The forgotten subspace projection matrix It is obtained through the following steps: Step 1: Obtain the text of the written work, mark the privacy information in the text, and construct a dataset accordingly; Step 2: Divide the dataset into a forget set Sf and a retention set Sr; Step 3: For each data point in the forget set Sf, input the data into the basic large language model and obtain the hidden state corresponding to each token in the Transformer layer. ,in, Indicates the sample index. This represents the token index. Then, average pooling or weighted pooling is applied to all hidden states within each Transformer layer to obtain the sample-level activation vector, represented as: ； Step 4: Repeat step 3 to obtain the sample-level activation vector corresponding to each data point in the forgetting set Sf, and use the sample-level activation vectors corresponding to all data points to obtain the representation matrix. ; Step 5: For the retained set Sr, repeat steps 3 and 4 to obtain the representation matrix corresponding to the retained set Sr. ; Step 6: For the representation matrix After centralization, singular value decomposition is performed, which is represented as: ； in, The left-hand unitary matrix after decomposition. The resulting diagonal matrix is the decomposed matrix. The decomposed right-hand unitary matrix, Let be the mean vector of the matrix; Step 7: According to Construct a base for preserving subspaces ; Step 8: Utilize the preserved subspace basis The projection matrix of the preserved subspace is obtained. , is represented as: ； Step 9: Based on the representation matrix and the reserved subspace projection matrix The purified representation matrix is obtained. , is represented as: ； in, It is the identity matrix; Step 10: Process the purified representation matrix Perform singular value decomposition, represented as: ； in, The left-hand unitary matrix after decomposition. The resulting diagonal matrix is the decomposed matrix. The right-hand unitary matrix after decomposition Step 11: According to Constructing the forgetting subspace basis ; Step 12: Utilize the forgotten subspace basis The projection matrix of the forgotten subspace is obtained. , is represented as: 。 3. The text privacy protection method based on subspace projection large language model forgetting as described in claim 2, characterized in that... The specific steps of step 8 are as follows: exist In the middle, before selecting by singular value energy In one direction, construct a preserved subspace basis. ,in, ； in, For the first matrix A singular value, It is a constant. Let be the number of singular values of the matrix. For the first matrix One singular value; The specific steps of step 12 are as follows: exist In the middle, before selecting by singular value energy One direction for constructing the forgetting subspace basis. ,in, ； in, For the first matrix A singular value, It is a constant.
4. The text privacy protection method based on subspace projection large language model forgetting as described in claim 1, characterized in that... In step 1, the privacy information in the text is marked through semantic retrieval or manual annotation.
5. A text privacy protection method based on subspace projection large language model forgetting as described in claim 1, characterized in that... The underlying model is Llama3-3b-Instruct or LlaMa3-8b-Instruct.
6. A text privacy protection system based on subspace projection large language model forgetting, characterized in that... The system specifically performs the following steps: For the text of the literary work to be protected, the text of the literary work to be protected is input into the large language model, and the output privacy-protected text is obtained. The large language model has at least one Transformer layer followed by a linear projection and recalibration layer, which utilizes a forgetting subspace projection matrix obtained based on the forgetting set and the retained set. For the original hidden state Perform linear projection and recalibration to obtain the hidden state. , is represented as: ； in, This is the hidden state after the transformation. It is in its original hidden state.
7. A text privacy protection system based on subspace projection large language model forgetting as described in claim 6, characterized in that... The forgotten subspace projection matrix It is obtained through the following steps: Step 1: Obtain the text of the written work, mark the privacy information in the text, and construct a dataset accordingly; Step 2: Divide the dataset into a forget set Sf and a retention set Sr; Step 3: For each data point in the forget set Sf, input the data into the basic large language model and obtain the hidden state corresponding to each token in the Transformer layer. ,in, Indicates the sample index. This represents the token index. Then, average pooling or weighted pooling is applied to all hidden states within each Transformer layer to obtain the sample-level activation vector, represented as: ； Step 4: Repeat step 3 to obtain the sample-level activation vector corresponding to each data point in the forgetting set Sf, and use the sample-level activation vectors corresponding to all data points to obtain the representation matrix. ; Step 5: For the retained set Sr, repeat steps 3 and 4 to obtain the representation matrix corresponding to the retained set Sr. ; Step 6: For the representation matrix After centralization, singular value decomposition is performed, which is represented as: ； in, The left-hand unitary matrix after decomposition. The resulting diagonal matrix is the decomposed matrix. The decomposed right-hand unitary matrix, Let be the mean vector of the matrix; Step 7: According to Construct a base for preserving subspaces ; Step 8: Utilize the preserved subspace basis The projection matrix of the preserved subspace is obtained. , is represented as: ； Step 9: Based on the representation matrix and the reserved subspace projection matrix The purified representation matrix is obtained. , is represented as: ； in, It is the identity matrix; Step 10: Process the purified representation matrix Perform singular value decomposition, represented as: ； in, The left-hand unitary matrix after decomposition. The resulting diagonal matrix is the decomposed matrix. The right-hand unitary matrix after decomposition Step 11: According to Constructing the forgetting subspace basis ; Step 12: Utilize the forgotten subspace basis The projection matrix of the forgotten subspace is obtained. , is represented as: 。 8. A text privacy protection system based on subspace projection large language model forgetting as described in claim 7, characterized in that... The specific steps of step 8 are as follows: exist In the middle, before selecting by singular value energy In one direction, construct a preserved subspace basis. ,in, ； in, For the first matrix A singular value, It is a constant. Let be the number of singular values of the matrix. For the first matrix One singular value; The specific steps of step 12 are as follows: exist In the middle, before selecting by singular value energy One direction for constructing the forgetting subspace basis. ,in, ； in, For the first matrix A singular value, It is a constant.
9. A text privacy protection system based on subspace projection large language model forgetting as described in claim 6, characterized in that... In step 1, the privacy information in the text is marked through semantic retrieval or manual annotation.
10. A text privacy protection system based on subspace projection large language model forgetting as described in claim 6, characterized in that... The underlying model is Llama3-3b-Instruct or LlaMa3-8b-Instruct.