Purification method and equipment for multi-modal unlearnable sample and storage medium
By preprocessing and training image and text samples separately, and combining them with an intramodal dispersion loss term, the problem of cleaning up unlearnable samples in multimodal modes is solved, achieving effective cleaning across modalities and restoring the learnability of the dataset.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-14
AI Technical Summary
Existing methods for cleaning up unlearnable samples are mainly limited to a single modality and cannot effectively handle multimodal unlearnable samples. Furthermore, existing methods suffer from high computational resource consumption and low time efficiency.
Multimodal cleanup is achieved by using JPEG compression to remove errors and minimize noise from image samples, calculating doubt scores to remove trigger words from text samples, and adding an intramodal dispersion loss term during training to push feature vectors apart.
It effectively restores the learnability of multimodal datasets, significantly improves the model's cleansing effect, expands the applicability of cleansing methods, and approaches the performance of clean data.
Smart Images
Figure CN121858993A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus and storage medium for purifying multimodal unlearnable samples. Background Technology
[0002] Unlearnable samples are a proactive data privacy protection strategy that injects carefully designed perturbations, imperceptible to the human eye, into the training data. This aims to prevent deep learning models from learning the generalized features of the data itself, instead encouraging them to learn spurious features created by the perturbations—also known as "shortcuts." The model will perform well on the training set but poorly on a clean test set. Unlearnable samples are used in modalities such as images, text, and audio, and are beginning to be applied to multimodal domains. Among them, Liu X et al. first proposed the problem of "multimodal unlearnable examples" in multi-step error minimization (MEM) (Liu X, Jia X, Xun Y, et al. Multimodalunlearnable examples: Protecting data against multimodal contrastive learning[C] / / Proceedings of the 32nd ACM International Conference on Multimedia.2024: 8024-8033.). By adding error minimization noise to the image modality and trigger words to the text modality, the model is induced to learn the relationship between noise and trigger words, thereby protecting the data itself from illegal use.
[0003] While unlearnable samples are often used as a data protection measure, they also pose a serious threat to model performance and security, potentially causing significant losses to a company's business interests. Therefore, researching techniques for sanitizing unlearnable samples is equally important.
[0004] Currently, the main methods for cleaning up unlearnable samples are as follows:
[0005] (1) Model-based training methods directly train the model on unlearnable datasets and learn generalized features from unlearnable samples by modifying the loss function and training strategy. The advantages of this method are that it is relatively simple to implement and has a wide range of applications, but it also has some disadvantages, such as requiring a lot of computing resources and having low time efficiency.
[0006] (2) Based on data preprocessing, some preprocessing operations are performed on the collected suspicious dataset before training the model, in an attempt to destroy potentially unlearnable samples and restore the learnability of the data. The advantage of this method is its low processing cost, but it also has some disadvantages, such as the need to balance the image quality and purification effect after processing.
[0007] In summary, existing cleanup methods all have certain limitations and can only process images as a single modality. Therefore, there is an urgent need for a new multimodal non-learnable sample cleanup system that can overcome the shortcomings of existing technologies and provide an effective and convenient cross-modal cleanup method. Summary of the Invention
[0008] Purpose of the invention: To address the above problems, the purpose of this invention is to provide a method, device, and storage medium for purifying multimodal unlearnable samples. By processing image and text modalities separately before training, and pushing apart feature vectors with distances less than a threshold within a single modality during training, spurious connections between sample pairs are removed, thereby purifying the dataset and restoring its learnability.
[0009] Technical solution: According to a first aspect of the present invention, a method for cleaning up multimodal unlearnable samples based on data preprocessing is provided, comprising the following steps:
[0010] Obtain a multimodal, non-learnable sample set containing image and text samples, and separate the image and text samples in the sample set;
[0011] Image samples are preprocessed to remove errors and minimize noise, resulting in cleaned image samples.
[0012] The text samples are preprocessed, including: calculating the doubt scores of words in the text, and deleting trigger words in the text based on the doubt scores to cut off the false association between image noise and text trigger words, so as to obtain the cleaned text samples;
[0013] The purified image and text samples are merged into a single dataset. The multimodal model is then iteratively trained using contrastive loss and intramodal dispersion loss to achieve sample purification.
[0014] Furthermore, the steps for preprocessing image samples include:
[0015] Perform JPEG compression on the image samples.
[0016] Furthermore, the steps for calculating the skepticism score of words in the text include:
[0017] The text samples are input into a pre-trained large language model to calculate the perplexity of the sentences;
[0018] Based on sentence perplexity, a doubt score is calculated for each word in the sentence, where the doubt score is used to characterize the likelihood that the word is a trigger word that establishes an association with image noise.
[0019] Furthermore, the large language model is the GPT-2 model.
[0020] Furthermore, the steps for removing trigger words from text based on suspicion scores include:
[0021] Remove words with a suspicion score higher than a preset threshold from the sentence; or, remove the word with the highest suspicion score from the sentence. 1 word was deleted from the sentence, among which It is an integer greater than or equal to 1.
[0022] Furthermore, the formula for calculating the intramodal dispersion loss is as follows:
[0023] ,
[0024] In the formula, This indicates the number of samples in the training batch. This indicates the preset interval hyperparameter. Indicates the first The feature vector of each sample This represents the l2 norm.
[0025] According to a second aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the above-described method for cleaning up multimodal unlearnable samples.
[0026] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method for cleaning up multimodal unlearnable samples.
[0027] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:
[0028] The advantage of this invention lies in its simultaneous processing of both image and text modalities, enabling it to handle multimodal unlearnable samples. To achieve this, the image is first compressed using JPEG to remove noise that minimizes errors. Then, a large language model is used to detect abnormal words in the text, removing shortcut trigger words. Furthermore, this invention alleviates the modality collapse problem by adding an intramodal dispersion loss term to push apart feature vectors that are too close within a single modality. Finally, the method of this invention effectively cleans up the perturbations added to image-text pairs by MEM methods, restoring the learnability of the dataset. This invention solves the problem that previous methods could only handle the image modality, expanding the applicability of unlearnable sample cleanup methods. Attached Figure Description
[0029] Figure 1 A flowchart for a method to clean up multimodal unlearnable samples;
[0030] Figure 2 This is a diagram illustrating the overall framework of this method;
[0031] Figure 3 This is the attention map of the CLIP model after training on datasets with different processing steps. Detailed Implementation
[0032] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.
[0033] In the following description, specific details such as target system architecture and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.
[0034] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0035] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0036] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0037] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0038] Combination Figure 1 and Figure 2 As shown in this embodiment, a method for cleaning up multimodal unlearnable samples includes the following steps:
[0039] Step 1: Obtain a multimodal non-learnable sample set containing image samples and text samples, and separate the image samples and text samples in the sample set.
[0040] Current methods for cleaning up unlearnable samples are designed for image modalities and do not consider multimodal scenarios. However, directly using existing cleaning methods to process multimodal unlearnable samples can only remove perturbations in the image modality and cannot handle the text modality. The cleaning effect is also quite mediocre, with most methods only recovering less than half the performance of those trained on clean data. In this example, the multimodal dataset is first split according to modality categories, with image samples in one category and text samples in another. Before training, image and text modalities are processed separately to remove spurious connections between sample pairs, thus cleaning the dataset and restoring its learnability.
[0041] Step 2: Perform data preprocessing on the image samples to remove errors and minimize noise, resulting in purified image samples.
[0042] Furthermore, the steps for preprocessing image samples include:
[0043] Perform JPEG compression on the image samples.
[0044] In one example, an image with MEM perturbation is JPEG compressed with a quality factor of 80. By utilizing the difference in frequency domain energy distribution, visual threshold, and nonlinear sparsification, invisible noise energy is preferentially discarded, thereby removing most of the error injected into it by MEM while maintaining image quality and minimizing noise.
[0045] Step 3 involves preprocessing the text samples, including calculating the doubt scores of words in the text and deleting trigger words from the text based on the doubt scores to sever the false association between image noise and text trigger words, thus obtaining the purified text samples.
[0046] Furthermore, the large language model is the GPT-2 model.
[0047] Furthermore, the steps for calculating the skepticism score of words in the text include:
[0048] The text samples are input into a pre-trained large language model to calculate the perplexity of the sentences;
[0049] Based on sentence perplexity, a doubt score is calculated for each word in the sentence, where the doubt score is used to characterize the likelihood that the word is a trigger word that establishes an association with image noise.
[0050] Furthermore, the steps for removing trigger words from text based on suspicion scores include:
[0051] Remove words with a suspicion score higher than a preset threshold from the sentence; or, remove the word with the highest suspicion score from the sentence. 1 word was deleted from the sentence, among which It is an integer greater than or equal to 1.
[0052] In one example, the specific implementation process of the text preprocessing stage includes the following steps:
[0053] Step 31: Calculate the initial perplexity value of the text using the Large Language Model GPT-2, denoted as . Treat a text sample as a sentence;
[0054] Step 32: Delete the words in the text one by one. Calculate and remove The perplexity value of the subsequent text ,Will minus You can get the word. doubt score .like Greater than the threshold Then the word is considered The text doesn't match its semantics; it might be a trigger word inserted by MEM. Remove it. Alternatively, [the text could be modified / removed]. Setting it to 10 will remove the 10 words with the highest suspicion scores from the sentence. Generally, the threshold can be set... The value is set to 0. The specific principle is as follows:
[0055] Given sentence The sentence includes Each word, GPT-2, with its autoregressive language model properties, is used for each position. Provide conditional probability distribution The perplexity value of a sentence. ) is defined as:
[0056] ,
[0057] Should Indicators measure the model's ability to model the language of a sentence as a whole: The lower the value, the smoother the sentence; if a certain word This is an abnormal trigger word; its introduction will lead to... A significant decrease, thus significantly improving overall PPL. Therefore, by calculating the removal The change in the subsequent confusion value can be quantified. The degree of suspicion.
[0058] Step 4: Use the cleaned image and text samples to perform contrastive learning training on the multimodal model. Add intramodal dispersion loss to the contrastive loss to push apart feature vectors that are too close in a single modality, thereby further improving the cleanup effect.
[0059] In one example, the CLIP model is used as a multimodal model. The CLIP model consists of two main encoders: an image encoder and a text encoder. Cleaned image and text samples are used as input to the CLIP model. The image encoder maps the image to a shared embedding space, and the text encoder maps the text samples to the shared embedding space, yielding the corresponding feature vectors. This includes image features and text features; a cosine similarity model is constructed using an intra-batch contrastive learning mechanism. A similarity matrix is used, and symmetric cross-entropy loss is employed to maximize the mutual information of positively matched image-text pairs and minimize the similarity of negatively matched samples to achieve semantic alignment. The similarity between feature vectors within a single modality is then determined. If the distance is less than a set threshold, a loss term is applied to push apart feature vectors that are too close together, thus alleviating the mode collapse problem.
[0060] Furthermore, the multimodal model is trained using contrastive learning, employing loss functions including contrastive loss and intramodal dispersion loss. Intramodal dispersion loss is used to increase the distance between feature vectors of different samples within the same modality. The formula for calculating intramodal dispersion loss is:
[0061] ,
[0062] In the formula, This indicates the number of samples in the training batch. This indicates the preset interval hyperparameter. Indicates the first The feature vector of each sample This represents the l2 norm. This regularization term, by explicitly penalizing excessive proximity between feature vectors, forces the preservation of variance and diversity in modal embeddings, effectively suppressing noise-induced local collapse, thereby improving the robustness and discriminability of the overall representation.
[0063] The formula for calculating the contrast loss is:
[0064] ,
[0065] In the formula, This indicates the number of samples in the training batch. Indicates the first Feature vectors of image samples Indicates the first Feature vectors of a text sample The cosine similarity between two vectors is represented by the following expression: Indicates temperature parameter, This represents an exponential function. The contrastive loss, through bidirectional contrastive learning, maximizes the similarity of matching image-text pairs and minimizes the similarity of non-matching pairs, enabling the model to learn to map semantically related images and text to the same feature space, thus achieving cross-modal retrieval.
[0066] To further demonstrate the effectiveness and superiority of the purification learning method for multimodal unlearnable samples described in this invention, the following examples are provided.
[0067] Using the Flickr8K dataset as both the training and testing sets, the multimodal model was trained on samples trained without data (Random), clean samples, MEM-3 generated samples, and samples processed with JPEG compression, BDR, Gray, adversarial training (AT), adversarial enhancement (AA), and the method of this invention. Table 1 shows the experimental results of each method on the Flickr8K dataset with MEM-3 perturbation. JPEG compression removes perturbations by discarding high-frequency information that is insensitive to the human eye; BDR quantizes high-bit images to low bits to discard perturbations; Gray converts the original image into a single-channel grayscale image containing only luminance information and no color; adversarial training actively introduces carefully designed perturbations during training; and adversarial enhancement combines effective data augmentation strategies with loss-maximizing adversarial enhancement methods to improve the model's robustness and generalization. Hit@10 and Medr were used as metrics to evaluate the model's training performance. Hit@10 represents the proportion of correct samples appearing in the top 10 results across all search results; Medr represents the median ranking of correct samples across all search results. A higher Hit@10 value and a lower Medr value indicate better model training. Table 1 shows that in the Flickr8K cross-modal retrieval task, the method described in this invention significantly outperforms other existing cleansing methods in both image-to-text and text-to-image directions. Hit@10 improves from 14.1 and 12.5 in the best comparison method AA to 22.1 and 19.9, respectively, while Medr also decreases to 56 and 62, close to the Clean model's 57 and 60. This indicates that the method described in this invention can effectively eliminate the effects of perturbations and almost completely restore the model's performance.
[0068] Table 1. Experimental results of each method on the Flickr8K dataset for handling MEM-3 perturbations.
[0069] Table 2 shows the experimental results of each method on the Flickr8K dataset for processing MEM-5 perturbations. The Flickr8K dataset was used as both the training and testing sets. The multimodal model was trained on clean samples, MEM-5 generated samples, and samples processed by JPEG compression, BDR, Gray, AT, AA, and the method of this invention, respectively. Hit@10 and Medr were used as metrics to evaluate the training performance of the models. The experimental results show that in the cross-modal retrieval task on Flickr8K, the method described in this invention achieves Hit@10 scores of 21.3 and 20.0 on Image-Text and Text-Image tasks, respectively, with Medr scores of 55 and 66. This significantly outperforms other comparative methods and narrows the gap with the clean model to within 2 percentage points, demonstrating near-upper-level cleanliness capabilities.
[0070] Table 2. Experimental results of each method on the Flickr8K dataset for handling MEM-5 perturbations.
[0071] like Figure 3The attention heatmaps of the images and text shown intuitively present the attention distribution of the model trained on (1) clean samples, (2) unlearnable samples, and samples processed by (4) adversarial training, (5) JPEG compression, (6) BDR, (7) Gray, (8) adversarial enhancement, and (3) the method described in this invention. Among them, Grad-CAM (Selvaraju RR, Cogswell M, Das A, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization[C] / / Proceedings of the IEEE internationalconference on computer vision. 2017: 618-626.) is used to visualize the attention map of the image, while integral gradient method (Sundararajan M, Taly A, Yan Q. Axiomatic attribution for deep networks[C] / / International conference on machine learning. PMLR, 2017: 3319-3328.) is used to present the attention features of the text, and the darker the color, the higher the degree of attention of the model. For images, model (1) focuses on the central region, which matches the caption content. In contrast, model (2), trained on MEM-3 generated samples, failed to accurately identify clear images because it only learned noise features. Models (3) to (8), trained on various cleaned samples, focused on the correct regions to varying degrees, with model (3) trained on cleaned samples using the method described in this invention being closest to model (1). Similarly, in text processing, different models focus on different keywords, and model (3) of this method is closest to the clean model (1). These visualization results demonstrate that the method described in this invention has a good effect on purifying MEM noise.
[0072] The purpose of this invention is to provide a method for cleaning up unlearnable samples based on multimodal processing. The advantage of this method is that it processes both image and text modalities simultaneously, thus handling unlearnable samples across multiple modalities. To achieve this, the image is first compressed using JPEG to remove noise that minimizes errors. Then, a large language model is used to detect abnormal words in the text, removing shortcut trigger words. Furthermore, this invention alleviates the modality collapse problem by incorporating an intramodal dispersion loss term to separate feature vectors that are too close within a single modality. Finally, this invention effectively cleans up the perturbations introduced into image-text pairs by the MEM method, restoring the learnability of the dataset. This multimodal unlearnable sample cleaning method based on data preprocessing overcomes the limitation of previous methods that could only handle the image modality, expanding the applicability of unlearnable sample cleaning methods.
[0073] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for cleaning up multimodal unlearnable samples.
[0074] It is understood that the beneficial effects of the electronic device in this embodiment can be referred to the beneficial effects of the above-described embodiment of the purification method for multimodal unlearnable samples, and will not be repeated here.
[0075] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for cleaning up multimodal unlearnable samples.
[0076] It is understood that the beneficial effects of the computer-readable storage medium in this embodiment can be referred to the beneficial effects of the embodiment of the method for cleaning up multimodal unlearnable samples, and will not be repeated here.
Claims
1. A method for cleaning up multimodal unlearnable samples based on data preprocessing, characterized in that, Includes the following steps: Obtain a multimodal, non-learnable sample set containing image and text samples, and separate the image and text samples in the sample set; Image samples are preprocessed to remove errors and minimize noise, resulting in cleaned image samples. The text samples are preprocessed, including: calculating the doubt scores of words in the text, and deleting trigger words in the text based on the doubt scores to cut off the false association between image noise and text trigger words, so as to obtain the cleaned text samples; The purified image and text samples are merged into a single dataset. The multimodal model is then iteratively trained using contrastive loss and intramodal dispersion loss to achieve sample purification.
2. The method for purifying multimodal unlearnable samples according to claim 1, characterized in that, The steps for preprocessing image samples include: Perform JPEG compression on the image samples.
3. The method for purifying multimodal unlearnable samples according to claim 1, characterized in that, The steps for calculating the skepticism score of words in a text include: The text samples are input into a pre-trained large language model to calculate the perplexity of the sentences; Based on sentence perplexity, a doubt score is calculated for each word in the sentence, where the doubt score is used to characterize the likelihood that the word is a trigger word that establishes an association with image noise.
4. The method for purifying multimodal unlearnable samples according to claim 3, characterized in that, The large language model is the GPT-2 model.
5. A method for purifying multimodal unlearnable samples according to claim 3 or 4, characterized in that, The steps for removing trigger words from text based on suspicion scores include: Remove words with a suspicion score higher than a preset threshold from the sentence; or, remove the word with the highest suspicion score from the sentence. 1 word was deleted from the sentence, among which It is an integer greater than or equal to 1.
6. The method for purifying multimodal unlearnable samples according to claim 1, characterized in that, The formula for calculating the intramodal dispersion loss is as follows: ， In the formula, This indicates the number of samples in the training batch. This indicates the preset interval hyperparameter. Indicates the first The feature vector of each sample This represents the l2 norm.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.