A multi-modal large language model passive forgetting method based on proxy anchor points

Abstract

The application discloses a multi-modal large language model passive forgetting method based on an agent anchor point, and aims at solving the problem that original private data cannot be accessed in a privacy compliance scene. First, a text-guided coarse-to-fine retrieval strategy is adopted, cross-modal feature alignment is utilized to accurately locate an agent anchor point which overlaps with a target semantic from a public data set, and a substitute supervision signal is constructed. Secondly, a double-constraint semantic isolation optimization is implemented. On one hand, a text-anchor semantic repulsion mechanism is introduced to cut off a visual-induced link of a target concept in a feature space, and accurate erasing is realized. On the other hand, a zero-space projection technology is utilized to strictly limit gradient updating in an orthogonal subspace which retains knowledge, and feature isotropic regularization is used to prevent manifold collapse. While completely forgetting sensitive concepts, the method effectively guarantees the general perception and reasoning ability of the model, and significantly reduces the risk of catastrophic forgetting.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision security technology, specifically relating to a passive forgetting method for multimodal large language models based on proxy anchors. Background Technology

[0002] Multimodal large language models, pre-trained on massive amounts of text and images, possess powerful cross-modal understanding and generation capabilities. However, they also internalize a large amount of sensitive personal information, posing risks of privacy leaks and copyright infringement. Machine forgetting technology aims to eliminate the influence of specific data or concepts from pre-trained models and is a key means to achieve model compliance. Existing mainstream machine forgetting methods generally rely on the original training data as optimization constraints or supervision signals. In practical high-security application scenarios, due to data privacy regulations and strict data retention policies, the storage and re-access of the original target image are often prohibited. This data inaccessibility directly renders conventional forgetting algorithms that rely on original data unenforceable in "passive" scenarios.

[0003] Existing passive forgetting techniques are primarily designed for discriminative models such as image classification, typically employing synthetic substitution signals or data-free knowledge distillation strategies. Generative multimodal large-scale models generate text through the mapping of visual encoding to the linguistic space, exhibiting deep visual-linguistic feature entanglement, significantly differing in structure from discriminative models. Existing passive methods cannot adapt to this generative coupling mechanism, struggling to accurately locate and extract semantic features closely associated with the target concept while completely blocking access to original visual data. How to construct effective feature substitution schemes in the absence of real visual samples, maintaining the structural integrity of the model's general generative capabilities while thoroughly erasing the target concept, is a pressing technical problem in the field of multimodal model security. Summary of the Invention

[0004] To address the aforementioned issues, this invention discloses a passive forgetting method for multimodal large language models based on proxy anchors. Addressing the limitation of missing target visual data in passive scenarios, it constructs semantically similar public proxy data as a fulcrum and combines geometric constraint optimization to achieve indirect erasure of target concepts and lossless maintenance of retained knowledge.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A passive forgetting method for multimodal large language models based on surrogate anchors includes the following steps:

[0007] S1. Set the forgetting target and data source: Determine the textual description of the target concept to be forgotten. And prepare a large-scale public dataset that does not contain specific target concepts. ;

[0008] S2. Adopt a coarse-to-fine strategy, starting from public datasets. The set of proxy anchors P that are semantically aligned with the target concept is retrieved. This step utilizes the semantic understanding capabilities of a large language model for initial screening and the feature alignment capabilities of a multimodal encoder for fine-grained matching.

[0009] S3. Construct the secure null projection matrix. Using the samples or general data selected in step S2 as proxies for preserving knowledge, calculate the feature covariance matrix and perform feature decomposition to construct the null projection matrix of the knowledge-preserving subspace. This is used to constrain subsequent gradient updates, ensuring that model updates are orthogonal to the knowledge-preserving feature space;

[0010] S4. Optimize model parameters on the proxy anchor set. Introduce text-anchor semantic repulsion loss during the optimization process. To sever the connection between visual features and target text, isotropic regularization of features is introduced. To prevent feature space collapse. Simultaneously, the gradient is strictly projected into the null space constructed in step S3;

[0011] S5. Calculate the original gradient based on the loss function constructed in step S4, and use the null projection matrix constructed in step S3. The original gradient is projected to obtain the projected gradient. The parameters of the multimodal large language model are updated using the projected gradient, ultimately yielding the model after forgetting specific concepts.

[0012] Furthermore, step S2 specifically includes:

[0013] Coarse-grained semantic filtering: Large language models (such as GPT-4 or Llama-3) are used as semantic filters. For each sample x in the public dataset, its metadata is compared with the target concept. Input LLM and determine whether the two are semantically related. The filtering formula is shown in Equation (1):

[0014] (1);

[0015] In equation (1), This represents the textual metadata or category label of sample x in a public dataset. An output of 1 indicates a relevant result, while an output of 0 indicates an unrelated result. Only samples with a result of 1 are retained to form the candidate pool. .

[0016] Fine-grained cross-modal matching: Utilizing the frozen visual encoder in a multimodal large language model and text encoder Images from the candidate pool and target concept text Mapping to a shared normalized feature space. Calculating the cosine similarity between visual embeddings and text embeddings. The formula is shown in equation (2):

[0017] (2);

[0018] in, express Norm. Candidate images are sorted in descending order based on similarity S, and the top K samples with the highest similarity are selected as surrogate anchors. Furthermore, to prevent overfitting by a single category feature, a category diversity constraint is introduced, limiting the number of samples belonging to the same semantic category in the final surrogate anchor set P to no more than a preset threshold. The final set P is the surrogate anchor set used for subsequent optimization.

[0019] Furthermore, step S3 specifically includes:

[0020] Using the surrogate anchor set or common retained data obtained in step S2 as a representation of retained knowledge, a null projection matrix is ​​constructed for the l-th linear layer of the model. The specific process is as follows:

[0021] Calculate the feature correlation matrix. Obtain the feature matrix input to this layer. Calculate the decentralized feature correlation matrix The calculation formula is shown in formula (3):

[0022] (3);

[0023] Where N is the number of samples and d is the dimension of the feature space.

[0024] Eigenvalue decomposition and spatial partitioning. For matrices... Eigenvalue decomposition is performed, and the decomposition method is shown in formula (4).

[0025] (4);

[0026] in, Representative matrix transpose, This is an eigenvalue diagonal matrix. An energy threshold is set. According to eigenvalues Size partitioning of feature space: preserving subspace : From eigenvalues > The corresponding eigenvector span represents the main direction of knowledge preservation; the safety null space : From eigenvalues The corresponding feature vector span represents a direction with extremely low correlation to the retained knowledge.

[0027] Constructing the projection matrix: Utilizing eigenvalues ​​in the covariance matrix of the retained knowledge that are less than a threshold The secure null basis matrix spanned by eigenvectors Constructing the projection matrix The construction method is shown in formula (5):

[0028] (5);

[0029] Representative matrix The transpose of is used to project the subsequent gradient vector into the safe null space.

[0030] Furthermore, step S4 specifically includes:

[0031] In the dual-constraint optimization objective, the total loss function is defined. The definition is as follows:

[0032] (6);

[0033] In equation (6), This refers to the loss of conventional tasks based on proxy anchors, such as the cross-entropy loss in autoregressive language modeling. It is text-anchor semantic exclusion loss. It is the feature isotropic regularization loss. and These are the hyperparameters that control the weights of the repulsion loss and the regularization loss, respectively. Textual description of the target concept The target text embedding vector is obtained by extracting the text using a text encoder.

[0034] Designed to cut off the visual features of the proxy anchor. Embedded with target concept text The semantic relationship between them. A hard constraint of cosine similarity with ReLU activation is adopted, and the calculation method is shown in equation (7):

[0035] (7);

[0036] In equation (7), N represents the number of proxy anchor samples in the current training batch. This represents the visual feature vector extracted from the i-th proxy anchor image by a multimodal large-model visual encoder. Textual description of the target concept The target text embedding vector is obtained by the text encoder. This represents the ReLU activation function, which is used to penalize cases where the cosine similarity is positive, imposing a hard orthogonal constraint.

[0037] The aim is to prevent dimensionality collapse in the feature space during erasure and to maintain the high-rank property of the feature manifold. The feature covariance matrix is ​​calculated. The Frobenius norm distance between the matrix I and the identity matrix is ​​calculated as shown in equation (8):

[0038] (8);

[0039] In equation (8), This represents the decentralized empirical covariance matrix of the visual features of the current batch of proxy anchor points. Indicates and Identity matrices of the same dimension denoted by Frobenius norm, used to measure the distance between matrices, and d denotes the dimension of the feature space. Represents the covariance matrix The j-th eigenvalue is used to characterize the variance of the feature distribution across different dimensions.

[0040] Furthermore, step S5 specifically includes:

[0041] The model parameters are updated using projected gradient descent. The actual update amount for the parameters of the l-th layer of the model is... The update rule is as follows: Calculate the original gradient, based on the total loss function. Calculate the original gradient with respect to the parameters. .

[0042] The projection matrix constructed using step S3 The safe gradient is obtained by right-multiplying the original gradient by the projection. :

[0043] (9);

[0044] In equation (9), Representative matrix transpose, Indicated based on the total loss function The original gradient vector obtained by calculating the parameters of the l-th layer model. This represents the null projection matrix of the l-th layer, used to filter out gradient components related to knowledge retention. This indicates that the eigenvalues ​​in the knowledge-preserving covariance matrix are less than a threshold. The safe null space basis matrix spanned by the eigenvectors.

[0045] The parameters can be updated using the projected safety gradient. During this process, the parameters of the visual encoder and the multimodal projection layer of the multimodal large language model are kept frozen and not updated. The parameter update formula is shown in equation (10):

[0046] (10);

[0047] In equation (10), The learning rate represents the parameter update. This represents the safety gradient vector after projection transformation, whose direction strictly lies within the knowledge-preserving null space. This represents the actual update amount of the parameters in the l-th layer model.

[0048] The beneficial effects of this invention are as follows:

[0049] This invention discloses a passive forgetting method for multimodal large language models based on surrogate anchors. First, it employs a text-guided coarse-to-fine retrieval strategy, utilizing the semantic reasoning capabilities of the large language model to remove irrelevant noise from public datasets. Then, through cross-modal feature alignment, it accurately locates surrogate anchors that highly overlap with the target concept semantically. This successfully constructs an effective alternative supervision signal without accessing the original private data, solving the problem of existing technologies being unable to implement forgetting in strict privacy-compliant scenarios. Second, this invention implements a dual-constraint semantic isolation optimization. On one hand, through a text-anchor semantic exclusion mechanism, it forcibly increases the geometric distance between the surrogate visual features and the target text embedding in the feature space, forcing the model to sever the causal link between visual input and the generation of specific sensitive text, achieving precise and thorough erasure of the target concept. On the other hand, it utilizes null space projection technology to construct a safe update subspace for preserving knowledge, strictly restricting gradient updates within this orthogonal space. Combined with isotropic feature regularization to prevent feature manifold collapse, this ensures that while the model forgets specific concepts, its general perception and reasoning capabilities for non-target concepts (i.e., preserving the structural integrity of knowledge) remain intact, significantly reducing the risk of catastrophic forgetting. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the overall architecture in an embodiment of the present invention;

[0051] Figure 2 This is a diagram illustrating the problem definition and input of the present invention;

[0052] Figure 3 A schematic diagram illustrating the selection of proxy anchor points for text guidance in this invention;

[0053] Figure 4 This is a schematic diagram of the dual-constraint semantic isolation of the present invention. Detailed Implementation

[0054] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0055] like Figure 1 As shown, this invention provides a passive forgetting method for multimodal large language models based on surrogate anchors. This method aims to achieve precise erasure of specific concepts and lossless maintenance of knowledge retention by constructing surrogate anchors and implementing dual geometric constraints, without accessing the original training data of the target concept. The specific implementation steps are as follows:

[0056] S1. Set the forgotten target and data source;

[0057] Please refer to Figure 2 , Figure 2 This is a problem definition and input diagram for the present invention.

[0058] First, clarify the goals and constraints of the machine forgetting task. Define a target concept set C to be removed from the pre-trained multimodal large language model. Only a textual description of the concept is required. For example, access to any image data containing the concept "Mango" is strictly prohibited. Prepare a large-scale, general-purpose, public image and text dataset. For example, subsets of CC3M, LAION-400M, or publicly available subsets of ImageNet-1k. This dataset serves only as a candidate pool for retrieving "surrogate anchors," does not need to contain specific target concept images, and must ensure that its data distribution has broad diversity to support coverage of various types of preserved knowledge.

[0059] S2, retrieve the proxy anchor point set P from coarse to fine;

[0060] Please refer to Figure 3 , Figure 3 A schematic diagram illustrating the selection of proxy anchor points for text guidance in this invention;

[0061] This step aims to construct alternative supervisory signals using public data. Since public datasets contain a large amount of noise irrelevant to the target, directly using them is computationally expensive and ineffective. Therefore, a two-stage retrieval strategy, from coarse to fine, is adopted:

[0062] Phase 1: Coarse-grained semantic filtering utilizes the powerful semantic reasoning capabilities of large language models as filters. For public datasets... For each sample x, extract its metadata meta(x), such as caption or category label. Then extract the target concept text. The metadata meta(x) is constructed as the Prompt input LLM, and the semantic relationship between the two is queried. The filtering and discrimination formula is shown in Equation (1):

[0063] (1);

[0064] In equation (1), This represents the textual metadata or category label of sample x in a public dataset. An output of 1 indicates a relevant sample, while an output of 0 indicates an irrelevant sample. This step quickly eliminates most irrelevant samples, narrowing the search scope to the candidate pool. .

[0065] Phase Two: Fine-grained Cross-modal Matching To find samples that can truly substitute for the target concept in the visual feature space, the frozen visual encoder of the MLLM to be forgotten is utilized. and text encoder Perform feature alignment. Align the images in the candidate pool. and target concept text Mapping to a shared normalized feature space, calculate the cosine similarity S( ). , As shown in equation (2):

[0066] (2);

[0067] in, express Norm. Candidate images are sorted in descending order based on similarity S, and the top K samples (e.g., K=2 to 8) are selected as surrogate anchors. To prevent overfitting due to overly homogeneous surrogate anchors, this invention introduces a category diversity constraint, limiting the number of anchors under the same semantic category to no more than 50% of the total. The final selected set P contains common images that highly overlap with the target concept in semantic space, serving as the fulcrum for subsequent optimization.

[0068] S3. Construct a secure zero-space projection matrix ;

[0069] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the dual-constraint semantic isolation of the present invention;

[0070] Using the surrogate anchor set or common retained data obtained in step S2 as a representation of retained knowledge, a null projection matrix is ​​constructed for the l-th linear layer of the model. The specific process is as follows:

[0071] Calculate the feature correlation matrix. Obtain the feature matrix input to this layer. Calculate the decentralized feature correlation matrix The calculation formula is shown in formula (3):

[0072] (3);

[0073] Where N is the number of samples and d is the feature dimension.

[0074] Eigenvalue decomposition and spatial partitioning. For matrices... Eigenvalue decomposition is performed, and the decomposition method is shown in formula (4).

[0075] (4);

[0076] in, Representative matrix transpose, This is an eigenvalue diagonal matrix. An energy threshold is set. (In this embodiment, The preferred range is 0.75 to 0.95 (e.g., 0.88), based on the characteristic value. Size partitioning of feature space: preserving subspace : From eigenvalues > The corresponding eigenvector span represents the main direction of knowledge preservation; the safety null space : From eigenvalues The corresponding feature vector span represents a direction with extremely low correlation to the retained knowledge.

[0077] Constructing the projection matrix: using the null space basis vectors Constructing the projection matrix The construction method is shown in formula (5):

[0078] (5);

[0079] Representative matrix The transpose of is used to project the subsequent gradient vector into the safe null space.

[0080] S4. Define the dual-constraint optimization objective;

[0081] In the dual-constraint optimization objective, the total loss function is defined. The definition is as follows:

[0082] (6);

[0083] In equation (6), This refers to the loss of conventional tasks based on proxy anchors, such as the cross-entropy loss in autoregressive language modeling. It is text-anchor semantic exclusion loss. It is the feature isotropic regularization loss. and These are the hyperparameters that control the weights of the repulsion loss and the regularization loss, respectively. In this embodiment, the weight hyperparameters are... The preferred value is 0.1, a weighted hyperparameter. The preferred value is 1.5.

[0084] Designed to cut off the visual features of the proxy anchor. Embedded with target concept text The semantic relationship between them. A hard constraint of cosine similarity with ReLU activation is adopted, and the calculation method is shown in equation (7):

[0085] (7);

[0086] In equation (7), N represents the number of proxy anchor samples in the current training batch. This represents the visual feature vector extracted from the i-th proxy anchor image by a multimodal large-model visual encoder. Textual description of the target concept The target text embedding vector is obtained by the text encoder. This represents the ReLU activation function, which is used to penalize cases where the cosine similarity is positive, imposing a hard orthogonal constraint.

[0087] The aim is to prevent dimensionality collapse in the feature space during erasure and to maintain the high-rank property of the feature manifold. The feature covariance matrix is ​​calculated. The Frobenius norm distance between the matrix I and the identity matrix is ​​calculated as shown in equation (8):

[0088] (8);

[0089] In equation (8), This represents the decentralized empirical covariance matrix of the visual features of the current batch of proxy anchor points. Indicates and Identity matrices of the same dimension denoted by Frobenius norm, used to measure the distance between matrices, and d denotes the dimension of the feature space. Represents the covariance matrix The j-th eigenvalue is used to characterize the variance of the feature distribution across different dimensions.

[0090] S5. Parameter update based on null projection;

[0091] The model parameters are updated using projective gradient descent. For the parameters of the l-th layer of the model... The update rule is as follows: Calculate the original gradient, based on the total loss function. Calculate the original gradient with respect to the parameters. .

[0092] The projection matrix constructed using step S3 The safe gradient is obtained by right-multiplying the original gradient by the projection. :

[0093] (9);

[0094] In equation (9), Indicated based on the total loss function The original gradient vector obtained by calculating the parameters of the l-th layer model. This represents the null projection matrix of the l-th layer, used to filter out gradient components related to knowledge retention. This indicates that the eigenvalues ​​in the knowledge-preserving covariance matrix are less than a threshold. The safe null space basis matrix spanned by the eigenvectors.

[0095] The parameters can be updated using the projected safety gradient. During this process, the parameters of the visual encoder and the multimodal projection layer of the multimodal large language model are kept frozen and not updated. The parameter update formula is shown in equation (10):

[0096] (10);

[0097] In equation (10), The learning rate represents the parameter update. This represents the safety gradient vector after projection transformation, whose direction strictly lies within the knowledge-preserving null space. This represents the actual update amount of the parameters in the l-th layer model.

[0098] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A passive forgetting method for multimodal large language models based on surrogate anchors, characterized in that: Specifically, the following steps are included: S1. Set the forgetting target and data source; determine the text description of the target concept to be forgotten. And prepare a large-scale public dataset that does not contain specific target concepts. ; S2. Retrieve the set of proxy anchor points from coarse to fine; Coarse-grained semantic filtering: Using a large language model as a semantic filter; for each sample x in the public dataset, its metadata is compared with the target concept. Input LLM and determine whether the two are semantically related. The filtering formula is shown in Equation (1): (1)； In equation (1), This represents the textual metadata or category label of sample x in a public dataset. An output of 1 indicates a relevance, and an output of 0 indicates no relevance; only samples with a judgment of 1 are retained to form the candidate pool. ; Fine-grained cross-modal matching: Utilizing the frozen visual encoder in a multimodal large language model and text encoder Images from the candidate pool and target concept text Mapped to a shared normalized feature space; Computational cosine similarity between visual embeddings and text embeddings The formula is shown in equation (2): (2)； in, express Norm; Candidate images are sorted in descending order based on similarity S, and the top K samples with the highest similarity are selected as surrogate anchors; In addition, to prevent overfitting of single-category features, a category diversity constraint is introduced, that is, the number of samples belonging to the same semantic category in the final surrogate anchor set P is limited to no more than a preset threshold; The final set P is the surrogate anchor set used for subsequent optimization. S3. Construct a secure null-space projection matrix; Using the surrogate anchor set or common retained data obtained in step S2 as a representation of retained knowledge, a null space projection matrix is ​​constructed for the l-th linear layer of the model; the specific process is as follows: Calculate the feature correlation matrix: Obtain the feature matrix input to this layer. Calculate the decentralized feature correlation matrix The calculation formula is shown in formula (3): (3)； Where N is the number of samples and d is the feature dimension; Eigenvalue decomposition and space partitioning: for matrices Eigenvalue decomposition is performed, and the decomposition method is shown in formula (4). (4)； in, Representative matrix transpose, The eigenvalue diagonal matrix is ​​used; an energy threshold is set. According to eigenvalues Size partitioning of feature space: preserving subspace : From eigenvalues > The corresponding eigenvector span represents the main direction of knowledge preservation; the safety null space : From eigenvalues The corresponding feature vector span represents a direction with extremely low correlation to the retained knowledge; Constructing the projection matrix: Utilizing eigenvalues ​​in the covariance matrix of the retained knowledge that are less than a threshold The secure null basis matrix spanned by eigenvectors Constructing the projection matrix The construction method is shown in formula (5): (5)； Representative matrix The transpose of is used to project the subsequent gradient vector into the safe null space; S4. Define the dual-constraint optimization objective; In the dual-constraint optimization objective, the total loss function is defined. The definition is as follows: (6)； In equation (6), This refers to the loss of conventional tasks based on proxy anchors, such as the cross-entropy loss in autoregressive language modeling; It is text-anchor semantic exclusion loss. It is the feature isotropic regularization loss. and These are the hyperparameters that control the weights of the repulsion loss and the regularization loss, respectively. Designed to cut off the visual features of the proxy anchor. Embedded with target concept text The semantic relationship between them is determined by using ReLU activation and cosine similarity hard constraint, calculated as shown in equation (7). (7)； In equation (7), N represents the number of proxy anchor samples in the current training batch. This represents the visual feature vector extracted from the i-th proxy anchor image by a multimodal large-model visual encoder. Textual description of the target concept The target text embedding vector is obtained by the text encoder. This represents the ReLU activation function, which is used to penalize only cases where the cosine similarity is positive, imposing a hard orthogonal constraint; The aim is to prevent dimensional collapse of the feature space during erasure and to maintain the high-rank property of the feature manifold; the feature covariance matrix is ​​calculated. The Frobenius norm distance between the matrix I and the identity matrix is ​​calculated as shown in equation (8): (8)； In equation (8), This represents the decentralized empirical covariance matrix of the visual features of the current batch of proxy anchor points. Indicates and Identity matrices of the same dimension denoted by Frobenius norm, used to measure the distance between matrices, and d denotes the dimension of the feature space. Represents the covariance matrix The j-th eigenvalue is used to characterize the variance of the feature distribution across different dimensions; S5. Parameter update based on null projection Updating model parameters using projective gradient descent: For the parameters of the l-th layer of the model The update rule is as follows: Calculate the original gradient, based on the total loss function. Calculate the original gradient with respect to the parameters. ; The projection matrix constructed using step S3 The safe gradient is obtained by right-multiplying the original gradient by the projection. : (9)； In equation (9), Indicated based on the total loss function The original gradient vector obtained by calculating the parameters of the l-th layer model. This represents the null projection matrix of the l-th layer, used to filter out gradient components related to knowledge retention. This indicates that the eigenvalues ​​in the knowledge-preserving covariance matrix are less than a threshold. The safe null space basis matrix spanned by the eigenvectors; The parameters are updated using the safety gradient after projection. During this process, the parameters of the visual encoder and the multimodal projection layer of the multimodal large language model are kept frozen and not updated. The parameter update formula is shown in Equation (10): (10)； In equation (10), The learning rate represents the parameter update. This represents the safety gradient vector after projection transformation, whose direction lies strictly within the knowledge-preserving null space. This represents the actual update amount of the parameters in the l-th layer model.

Images