A multi-modal defense method based on fast-slow thinking cooperation and online update

Abstract

This invention discloses a multimodal defense method based on fast-slow thinking collaboration and online updates, belonging to the field of multimodal machine learning and artificial intelligence security technology. First, the teacher model extracts global features of the current image-text pair, calculates the image-text consistency confidence score, and directly outputs the judgment result or marks it as suspicious. Second, the student model extracts the image patch feature set of the suspicious sample pair, and calculates the optimal transmission distance with the text lexical feature set of each text in the candidate text queue maintained by the momentum text encoder, determining the best matching text and generating a cleaned sample pair. Finally, the student model extracts multi-level features of the cleaned sample pair, calculates the loss function, updates the student model parameters, and synchronizes the updated parameters to the teacher model and the momentum text encoder. This invention significantly improves computational efficiency while maintaining discrimination accuracy through fast-slow thinking collaboration, and continuously enhances the model's ability to resist poisoning attacks over time through online updates.

Description

Technical Field

[0001] This invention belongs to the field of multimodal machine learning and artificial intelligence security technology, specifically involving a multimodal defense method based on fast and slow thinking collaboration and online updates. Background Technology

[0002] Multimodal pre-trained models acquire cross-modal semantic alignment capabilities through contrastive learning of massive image-text pairings, and have become the foundational models for downstream tasks such as image-text matching and discrimination, cross-modal retrieval, zero-shot classification, and content moderation. In practical scenarios such as the automatic construction of large-scale image-text datasets, image-text consistency verification on content platforms, and continuous expansion of multimodal training data, image-text pairs often arrive continuously in a streaming manner, requiring the system to have three capabilities: first, to output matching decisions in real time for each arriving image-text pair to support downstream applications; second, to learn online from continuously arriving data, so that the model's representation capabilities continuously improve with data accumulation; and third, to identify and resist potentially poisoned image-text pairs that may be mixed into the data stream, avoiding malicious samples from contaminating the model.

[0003] Data poisoning attacks mainly fall into two categories: First, targeted data poisoning attacks, where attackers inject target images with adversarial text descriptions into the data stream, causing the model to misclassify the target image as the attacker's specified category during inference. A stable attack can be achieved by poisoning only one ten-thousandth of the data. Second, backdoor attacks, where attackers embed visible or invisible triggers into training images, causing images carrying the same triggers to be systematically misclassified. The attack effect can be triggered with a poisoning ratio as low as one-thousandth. More seriously, attack patterns are constantly evolving, and static, one-time data cleaning is insufficient to cope with ever-changing threats.

[0004] Existing image-text matching and poisoning defense methods mainly include: image-text pairing matching methods based on global semantic features, which re-select paired text for each image based on the global similarity between the image and text pools to break the poisoning association; and safety separation methods based on global features, which divide the data into a safe set and a high-risk set and then apply cross-modal contrastive learning only to the safe set. However, the above methods have the following core limitations:

[0005] First, the discrimination granularity is coarse. Existing methods mainly rely on the global features of images and text for consistency discrimination, which is insufficient for perceiving subtle poisoning signals or local semantic inconsistencies within the image-text pair, resulting in poor identification and correction of poisoned samples.

[0006] Secondly, the processing strategies are rigid, making it difficult to balance efficiency and accuracy. Existing methods apply the same defense process to all samples indiscriminately: if lightweight global feature methods are used uniformly, the accuracy in distinguishing difficult and poisoned samples is insufficient; if fine-grained analysis methods are used uniformly, the computational cost is high and it is difficult to meet the real-time requirements of streaming scenarios. In reality, the vast majority of streaming data consists of regular image-text pairs, with only a small number being difficult or poisoned samples, which should be treated differently.

[0007] Third, discrimination, defense, and model updates are disconnected. Existing methods treat image-text matching discrimination, poisoning defense, and model training as independent stages. The fine-grained analysis results produced during the defense process are only used for the current sample filtering and cannot serve as high-value supervisory signals to continuously feed back into the model. As a result, the model's discrimination and defense capabilities cannot evolve autonomously with the arrival of data.

[0008] Fourth, there is a lack of engineering mechanisms for streaming scenarios. Existing methods are mostly designed on static datasets and lack key engineering mechanisms for streaming online scenarios, such as maintaining the consistency of the candidate feature queue under continuous encoder updates, protecting the stability of the discriminative model under frequent parameter updates, and knowledge accumulation and forgetting resistance for high-value and difficult samples in streaming scenarios. Directly transferring these methods to streaming scenarios will lead to systemic problems such as representation drift, inference jitter, and catastrophic forgetting.

[0009] Therefore, there is an urgent need for a novel multimodal image-text discrimination method that can organically integrate task reasoning, poisoning prevention, and online model updates, and maintain system stability in streaming scenarios. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of existing technologies, such as coarse discrimination granularity, rigid strategies, disconnect between defense and updates, and lack of flow engineering mechanisms. Inspired by the dual-system theory of human cognition, this invention provides a multimodal defense method based on fast and slow thinking collaboration and online updates.

[0011] The present invention adopts the following technical solution:

[0012] Firstly, this invention proposes a multimodal defense method based on fast and slow thinking collaboration and online updates, comprising the following steps:

[0013] S1. Use the teacher model to extract the global image features and global text features of the current image-text pair, and calculate the image-text consistency confidence score. If the confidence score is higher than the first threshold or lower than the second threshold, output a matching decision or a non-matching decision, and end the processing of the current image-text pair. Otherwise, mark the image-text pair as a suspicious sample pair.

[0014] S2. Use the student model to extract the image block feature set of the suspicious sample pair, and calculate the optimal transmission distance with the text word feature set of each text in the candidate text queue maintained by the momentum text encoder to determine the best matching text; if the best matching text is consistent with the original text, the original image-text pair is used as the cleaned sample pair; if they are inconsistent, the original image-text pair is determined to be poisoned, and the image-text pair is reconstructed with the best matching text as the cleaned sample pair.

[0015] The student model has the same structure as the teacher model, both containing an image encoder and a text encoder; the momentum text encoder has the same structure as the text encoder in the student model.

[0016] S3. Using the student model, extract the image block feature set and text word feature set of the cleaned sample pair, as well as the image global feature and text global feature, calculate the loss function, update the parameters of the student model, and synchronize the updated parameters to the teacher model and the momentum text encoder.

[0017] Furthermore, the image-text consistency confidence score is calculated based on the cosine similarity between the global image features and the global text features of the current image-text pair.

[0018] Furthermore, the first threshold and the second threshold are adaptively determined based on the quantiles of the confidence distribution of all text-image pairs in the current stage.

[0019] Furthermore, in S2, the calculation process for the optimal transmission distance includes:

[0020] The image patch feature set of suspicious sample pairs extracted by the student model is used as the source distribution, and the text word feature set of each text in the candidate text queue is used as the target distribution. A similarity matrix is ​​constructed using the cosine similarity between the image patch features and the text word features, and the complement of the similarity matrix is ​​used as the cost matrix. The optimal transmission problem with entropy regularization is solved, and the transmission cost is used as the optimal transmission distance.

[0021] Furthermore, the loss function includes a weighted combination of contrastive semantic loss and cross-modal fine-grained alignment loss; the contrastive semantic loss is calculated based on the image global features and text global features of the sanitized sample pair, and the cross-modal fine-grained alignment loss is calculated based on the optimal transmission distance between the image patch feature set and the text lexical feature set of the sanitized sample pair.

[0022] Furthermore, in the process of calculating the optimal transmission distance between the image patch feature set and the text word feature set of the purified sample pair, the image patch feature set of the purified sample pair extracted by the student model is used as the source distribution, and the text word feature set of the purified sample pair extracted by the student model is used as the target distribution. A similarity matrix is ​​constructed using the cosine similarity between the image patch features and the text word features, and 1 is subtracted from the similarity matrix to obtain the cost matrix. The optimal transmission problem with entropy regularization is solved to obtain the transmission cost as the optimal transmission distance.

[0023] Furthermore, the process of synchronizing the updated parameters to the teacher model and the momentum text encoder includes:

[0024] The updated student model parameters are updated using an exponential moving average, and the teacher model does not receive gradient backpropagation.

[0025] The parameters of the momentum text encoder in the updated student model are updated using an exponential moving average.

[0026] Furthermore, the candidate text queue includes a set of lexical features obtained by encoding the original text of the current suspicious sample pair with the momentum text encoder, and a set of lexical features of the text encoded with the momentum text encoder in historical batches. The candidate text queue is maintained according to a first-in-first-out mechanism. After each training batch is completed, the set of lexical features obtained by encoding all texts in this batch with the momentum text encoder is appended to the end of the queue. If the queue capacity is full, the earliest entry record of the same quantity is eliminated.

[0027] Furthermore, the purified sample pairs processed by S2 and their priority scores are stored in the priority experience replay buffer pool, and historical samples are sampled from the buffer pool according to the priority scores and mixed with the current batch data for training each time the model is updated.

[0028] Furthermore, the priority score is determined based on the optimal transmission distance calculated in the slow thinking step S2 for the purified sample pair.

[0029] Secondly, the present invention provides a system for implementing the above-mentioned multimodal defense method based on fast and slow thinking collaboration and online updates, comprising:

[0030] The Quick Thinking subsystem includes a teacher model, which is used to calculate the consistency confidence of text-image pairs arriving in streaming mode, and output a match decision, a non-match decision, or mark the text-image pair as a suspicious sample pair.

[0031] The slow-thinking subsystem, which includes a student model, is used to process suspicious sample pairs marked by the fast-thinking subsystem into clean sample pairs by combining a candidate text queue maintained by the momentum text encoder.

[0032] The online update subsystem is used to calculate the loss function using the cleaned sample pairs obtained from the student model and the slow thinking subsystem, update the parameters of the student model, and synchronize the updated parameters to the teacher model and the momentum text encoder.

[0033] The beneficial effects of this invention are:

[0034] (1) This invention utilizes a teacher model to perform lightweight global feature matching on most regular image-text pairs and directly outputs a matching or non-matching decision to meet the real-time requirements of streaming scenarios; it only triggers a slow thinking step for suspicious sample pairs with confidence levels between two thresholds and performs deep reasoning through optimal transmission fine-grained matching, thereby achieving accurate identification of poisoned samples with limited computing resources.

[0035] (2) The present invention uses the image block feature set extracted by the student model and the text word feature set of each text in the candidate text queue to calculate the optimal transmission distance and perform comparative fine-grained matching. It can capture the subtle local semantic inconsistencies between the image and text pairs. Compared with the traditional global feature method, it significantly improves the perception of hidden poisoning signals such as targeted poisoning and backdoor attacks, and improves the detection rate of poisoned samples.

[0036] (3) This invention utilizes the student model to extract global and local features of the cleaned sample pairs to calculate the loss function, continuously updates the student model parameters, and synchronizes the updated parameters to the teacher model and momentum text encoder. As data streams continue to arrive, the discrimination ability of the teacher model and the fine-grained alignment ability of the student model are continuously enhanced, making it suitable for online discrimination, poisoning prevention, and continuous learning scenarios of large-scale image and text data streams. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating a multimodal defense method based on fast and slow thinking collaboration and online updates;

[0038] Figure 2 This is a flowchart illustrating the process of calculating fine-grained similarity scores using the slow-thinking subsystem.

[0039] Figure 3 This is a schematic diagram of the calculation process of the online update subsystem;

[0040] Figure 4 This is a schematic diagram of the key mechanisms of the online update subsystem;

[0041] Figure 5 This is a schematic diagram of a multimodal defense system based on fast and slow thinking collaboration and online updates. Detailed Implementation

[0042] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0043] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0044] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0045] This invention proposes a multimodal defense method based on fast and slow thinking collaboration and online updates, which is designed for streaming image-text pairs. ,in Represents a picture. The text description represents the corresponding image.

[0046] This invention comprises three subsystems: a fast thinking subsystem, a slow thinking subsystem, and an online update subsystem. The fast thinking subsystem includes a teacher model, and the slow thinking subsystem includes a student model. The teacher model and the student model have the same structure and both include an image encoder. With text encoder The online update subsystem includes a momentum version model, maintains a dynamic candidate text queue, a momentum text encoder, and a priority experience replay buffer pool.

[0047] Both the teacher model and the student model of this invention are multimodal pre-trained models, including an image encoder and a text encoder. The teacher model version is denoted as... , Its parameters are The student model version is denoted as , Its parameters are The momentum version only needs to maintain one text encoder, denoted as... Its parameters are It is specifically designed for use in the candidate queue.

[0048] The three subsystems described above are used to perform real-time matching and data poisoning defense on continuously arriving image-text pairs, thus mitigating the threat of data poisoning. Data poisoning threat is defined as: attackers injecting poisoned image-text pairs into the data stream; targeted poisoning pairs the target image with adversarial text; and backdoor attacks embed triggers in the image along with adversarial text. The system needs to output one of three judgments for each arriving image-text pair—match, mismatch, or suspected poisoning / mismatch—without explicitly knowing which samples are poisoned.

[0049] like Figure 1 As shown, the present invention mainly includes the following steps:

[0050] S1. For each image-text pair to be discriminated in the streaming multimodal data input. The fast thinking subsystem uses the teacher model version to extract global features of the image and global features of the text, and calculates the confidence level of image-text consistency.

[0051] The Quick Thinking subsystem corresponds to the intuitive judgment humans make about routine things. Its characteristics are closed-loop discrimination, no external references, and millisecond-level output.

[0052] For the input image Image encoder in the teacher model version Output image patch feature set and global features ,in, Indicates the length and width of the image. Dimensions representing features;

[0053] For input text The text encoder in the teacher model version Output text lexical feature set and global features ,in, Represents the length of the text sequence.

[0054] In this embodiment, the image encoder and text encoder can be implemented using existing models in the field, and global features are obtained by removing the average pooling layer.

[0055] In this embodiment, the confidence score is calculated using global feature cosine similarity, as shown in the following formula:

[0056]

[0057] in, These are the global image features and global text features calculated by the teacher model version, respectively. To express modulo, It is the i-th image-text pair The confidence level of consistency between the text and images.

[0058] S2, threshold adaptive.

[0059] The system performs a quick-thinking judgment based on the confidence level: if the confidence level is higher than the first threshold, a matching decision is directly output and the process terminates; if the confidence level is lower than the second threshold, a non-match decision is directly output and the process terminates; if the confidence level is between the first and second thresholds, the image-text pair is marked as a suspicious sample pair, triggering the slow-thinking subsystem.

[0060] In this embodiment, a first threshold is set. Second threshold The thresholds are adaptively determined based on the high quantile (e.g., 80%) and low quantile (e.g., 30%) of the confidence distribution for all image-text pairs in the current training phase, respectively, to avoid threshold failure caused by data distribution drift. Generally, the vast majority of regular image-text pairs are processed at this step, with only a few suspicious sample pairs entering the slow-thinking phase.

[0061] S3. For suspicious sample pairs that trigger slow thinking, the slow thinking subsystem calculates fine-grained similarity scores.

[0062] The slow-thinking subsystem uses a student model version of the image encoder to extract the image patch feature set of the image; using the candidate text queue as a reference space, it calculates the optimal transmission distance for the image patch feature set and the text word feature set of each candidate text in the candidate text queue, and uses the obtained distance as the fine-grained matching cost to obtain the fine-grained similarity score.

[0063] The slow-thinking subsystem corresponds to the reasoning process humans employ when facing difficult problems, involving external references and in-depth comparisons. Its characteristics include open-ended reasoning, invoking a comparative reference space, and drawing conclusions based on fine-grained comparisons. This invention's slow-thinking subsystem is constructed using the Optimal Transport Theory (OT) framework. The core problem of the OT framework is: given a source distribution... With target distribution In order to satisfy each source point The output is equal to its supply and the output of each target point. Given the marginal constraint that the received quantity equals its demand and the constraint that the transmitted quantity is non-negative, find the transmission scheme that minimizes the total transmission cost. The transmission scheme is determined by the transmission matrix. Description, each element in the matrix Indicates from the source point To the target point The amount of transported; the transmission cost is determined by the cost matrix. Description, in which elements Indicates from the source point To the target point The unit cost of transportation. The total transmission cost is the transmission matrix. With cost matrix The sum of element-wise products.

[0064] like Figure 2 As shown, suspicious images that trigger slow thinking. Image encoder from student model version Extract its image patch feature set For each text in the candidate queue Text encoder using momentum version Extract its text lexical feature set Image patch feature set As the source distribution, and using the text lexical feature set As the target distribution, based on image patch features Text lexical features Construct a similarity matrix using cosine similarity between the two and will As a cost matrix Solving the optimal transmission problem with entropy regularization yields the transmission cost. :

[0065]

[0066]

[0067]

[0068] by As a suspicious image With candidate text in the candidate queue The fine-grained similarity score indicates that the higher the score, the more semantically similar the two sentences are.

[0069] In one specific embodiment of the present invention, the reference space is obtained through a dynamic candidate text queue. To construct, the candidate text queue is constructed by a momentum version of the text encoder. Output features are maintained using a first-in, first-out (FIFO) mechanism: After each training batch is completed, the set of text lexical features obtained by encoding all texts in this batch using the momentum version of the text encoder is appended to the end of the queue; if the queue is full, the earliest enqueued record of equal quantity is discarded. The queue capacity is set to a fixed value. This ensures that the reference space has sufficient semantic coverage. Furthermore, suspicious images... Corresponding text It also needs to be included in the candidate text queue, and the text lexical feature set obtained by encoding using the momentum version of the text encoder is also used.

[0070] The introduction of the momentum version of the text encoder solves the representation drift problem of the candidate queue under continuous encoder parameter updates: if the student encoder is used directly to produce queue features, the early enqueued features and the current query image features are in different representation spaces, which will affect the reliability of OT matching.

[0071] S4. Based on the fine-grained similarity score, determine the best matching text for the suspicious image from the candidate text queue, and output the slow-thinking decision result and generate clean sample pairs according to the following rules.

[0072] If the best matching text is the original text Output matching decisions and treat the original text pairs as cleaned sample pairs. ;

[0073] If the best matching text matches the original text If there is a discrepancy, a suspected poisoning / mismatch judgment is output, and the best matching text is used as the reconstructed paired text. The reconstructed image-text pairs are used as purification sample pairs. .

[0074] This reasoning method, which involves putting suspicious images back into the candidate pool for re-comparison and matching, is fundamentally different from the closed-ended direct judgment of fast thinking. Fast thinking is an intuitive judgment of the current pair, while slow thinking is a comparative reasoning based on multiple candidates.

[0075] S5. Use purified samples to update the online subsystem.

[0076] like Figure 3 As shown, the loss function is first defined. This invention uses a slow loss function for the student model version of the text encoder. and image encoder Update accordingly. The loss function includes:

[0077] (5-1) Contrastive semantic loss :

[0078] Each purified sample pair formed after the slow-thinking reconstruction Using a student model version of the text encoder and image encoder Extract global features from the image respectively With text global features Global semantic alignment is achieved by employing image-text contrast learning loss:

[0079]

[0080]

[0081]

[0082] Where N is the number of clean sample pairs, It is a purified sample pair In Global features of the student version image It is a purified sample pair In Global features of the student version image It is a purified sample pair In Global features of the student version text It is a purified sample pair In Global features of the student version text It's a temperature over-parameter. It is the image-to-text loss. It is the loss from text to image.

[0083] (5-2) Cross-modal fine-grained alignment loss :

[0084] Each purified sample pair formed after the slow-thinking reconstruction Using a student model version of the text encoder and image encoder Extract image patch feature sets separately With text lexical feature set ,in It is an image patch feature. It is text lexical features; it is a set of image patch features. As the source distribution, and using the text lexical feature set As the target distribution, based on image patch features Text lexical features Construct a similarity matrix using cosine similarity between the two and will As a cost matrix Solving the optimal transmission problem with entropy regularization yields the transmission cost as the alignment loss for the cleaned sample pair:

[0085]

[0086]

[0087]

[0088] (5-3) Overall Loss :

[0089]

[0090] in, and and represent hyperparameters that control the contrastive semantic loss and the cross-modal fine-grained alignment loss, respectively. For ease of description, the above... and Using only one cleaned sample Taking this as an example, the average value of the same batch of purified sample pairs is calculated during actual operation.

[0091] (5-4) Update the subsystem online based on the total loss:

[0092] like Figure 4 As shown, the present invention designs the following key mechanisms:

[0093] At each training step, the student model version parameters based on Parameters are updated via gradient backpropagation, and EMA synchronization is immediately performed on the teacher model after the update. Specifically, the updated student model version parameters are synchronized. Update teacher model version parameters after performing exponential moving average. , means as follows:

[0094]

[0095] in, It is a sliding parameter, with a value between 0.99 and 0.999.

[0096] The teacher model does not accept gradient backpropagation. The FastThink subsystem always uses the teacher model for discrimination, which brings three engineering advantages: the teacher model parameters change smoothly, the FastThink output is stable and does not jitter; abnormal gradients caused by a single batch of poisoned samples are mitigated. The signal is strongly attenuated before reaching the teacher, so the disturbance to fast thinking can be ignored; students and teachers form a natural division of labor in the parameter space, with keen response and stable output, which physically corresponds to the binary structure of fast and slow thinking.

[0097] Using the updated student model version parameters Text encoder parameters in According to momentum coefficient The momentum version of the text encoder parameters is updated using EMA, resulting in a smoother change in the parameters, as shown below:

[0098]

[0099] in, Indicates the student model version parameter The text encoder parameters in the model include a momentum parameter, m, which takes a value between 0.99 and 0.999, thus ensuring that features within the queue are reliably matched in a slowly evolving unified representation space.

[0100] Using the updated momentum version of the text encoder parameters The text lexical feature set obtained by encoding the current batch of text is appended to the candidate queue. end, if The capacity is full, and the oldest record for the number of eliminated teams is now set.

[0101] In this embodiment, a buffer pool is maintained to store the purified sample pairs corresponding to the image-text pairs processed by slow thinking, which are used to update the subsystem according to preset rules. For example, each time the model is updated, historical samples are sampled from the buffer pool according to priority and probability and mixed with the current batch for training. This allows the poisoning pattern and difficult matching rules to be continuously learned and strengthened, combating the catastrophic forgetting problem in online learning scenarios and achieving a defensive evolution effect of discovery once and lifelong immunity.

[0102] This invention utilizes the collaborative operation of the three subsystems mentioned above. The poisoning patterns and fine-grained alignment knowledge discovered through slow thinking are continuously embedded into the student model parameters and synchronized to the teacher model via exponential moving average. This allows the discriminative ability of the fast thinking subsystem to autonomously evolve with the arrival of data. This invention can be applied to various downstream tasks, such as edge-to-edge processing (sampling, discrimination, cleaning, and training) of large-scale image and text training data; consistency verification of user-uploaded images and text on content platforms; secondary ranking and credibility filtering of cross-modal retrieval results; continuous learning and poisoning defense of deployed multimodal models, etc.

[0103] It should also be noted that the multimodal defense method based on fast-slow thinking collaboration and online updates in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a multimodal defense system based on fast-slow thinking collaboration and online updates, corresponding to the multimodal defense method based on fast-slow thinking collaboration and online updates provided in the above embodiments, such as... Figure 5 As shown, it includes:

[0104] The Fast Thinking subsystem includes a teacher model, which extracts global image features and global text features from streaming image-text pairs and calculates the image-text consistency confidence score. If the confidence score is higher than a first threshold or lower than a second threshold, a match or non-match decision is directly output; if it is between the two thresholds, the image-text pair is marked as a suspicious sample and the Slow Thinking subsystem is triggered.

[0105] The slow-thinking subsystem includes a student model. This student model extracts image patch feature sets from suspicious sample pairs and compares them with the lexical feature sets of each text in a candidate text queue maintained by the momentum text encoder through optimal transmission fine-grained matching to determine the best-matching text. If the best match matches the original text, the original sample pair is output as the cleaned sample pair; otherwise, it is determined to be poisoned, and the image-text pair reconstructed from the best-matching text is output as the cleaned sample pair.

[0106] The online update subsystem is used to calculate the loss function using cleaned sample pairs processed by the student model and the slow-thinking subsystem, update the parameters of the student model, and synchronize the updated parameters to the teacher model and the momentum text encoder. This subsystem contains three key mechanisms:

[0107] Momentum Text Encoder: It has the same structure as the text encoder of the student model. Its parameters are updated by exponential moving average of the parameters of the student model's text encoder. It is specifically used to generate features in the candidate text queue, ensuring that the text features entering the queue at different times are in a consistent representation space, thus solving the problem of inconsistent queue features caused by encoder parameter drift in online scenarios.

[0108] Teacher-student exponential moving average: Maintain a teacher model with the same structure as the student model. The teacher model parameters are obtained by exponential moving average of the student model parameters and do not receive gradient backpropagation. The fast thinking subsystem always uses a smooth and stable teacher model for discrimination, avoiding the fast thinking output jitter caused by frequent parameter updates of the student model, and has natural robustness to abnormal gradient updates caused by a single batch of poisoned samples.

[0109] Priority Experience Replay Buffer Pool: High-value samples processed by slow thinking and their priority scores are stored in the buffer pool. Each time the model is updated, historical samples are sampled from the buffer pool according to priority probability and mixed with the current batch for training. This allows the poisoning mode and difficult matching rules to be continuously learned and reinforced, combating the catastrophic forgetting problem in online learning scenarios and achieving a defensive evolution effect of discovery once and lifelong immunity.

[0110] For the system implementation, since it basically corresponds to the method implementation, the relevant parts can be referred to in the description of the method implementation. The implementation methods of each subsystem will not be repeated here.

[0111] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0112] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multimodal defense method based on fast and slow thinking collaboration and online updates, characterized in that, Includes the following steps: S1. Use the teacher model to extract the global image features and global text features of the current image-text pair, and calculate the image-text consistency confidence score. If the confidence score is higher than the first threshold or lower than the second threshold, output a matching decision or a non-matching decision, and end the processing of the current image-text pair. Otherwise, mark the image-text pair as a suspicious sample pair. S2. Use the student model to extract the image block feature set of the suspicious sample pair, and calculate the optimal transmission distance with the text word feature set of each text in the candidate text queue maintained by the momentum text encoder to determine the best matching text; if the best matching text is consistent with the original text, the original image-text pair is used as the cleaned sample pair; if they are inconsistent, the original image-text pair is determined to be poisoned, and the image-text pair is reconstructed with the best matching text as the cleaned sample pair. The student model has the same structure as the teacher model, both containing an image encoder and a text encoder; the momentum text encoder has the same structure as the text encoder in the student model. S3. Using the student model, extract the image block feature set and text word feature set of the cleaned sample pair, as well as the image global feature and text global feature, calculate the loss function, update the parameters of the student model, and synchronize the updated parameters to the teacher model and the momentum text encoder.

2. The multimodal defense method based on fast and slow thinking collaboration and online updates according to claim 1, characterized in that, The image-text consistency confidence score is calculated based on the cosine similarity between the global image features and the global text features of the current image-text pair.

3. The multimodal defense method based on fast and slow thinking collaboration and online updates according to claim 1, characterized in that, The first threshold and the second threshold are adaptively determined based on the quantiles of the confidence distribution of all text-image pairs in the current stage.

4. The multimodal defense method based on fast and slow thinking collaboration and online updates according to claim 1, characterized in that, In S2, the calculation process for the optimal transmission distance includes: The image patch feature set of suspicious sample pairs extracted by the student model is used as the source distribution, and the text word feature set of each text in the candidate text queue is used as the target distribution. A similarity matrix is ​​constructed using the cosine similarity between the image patch features and the text word features, and the complement of the similarity matrix is ​​used as the cost matrix. The optimal transmission problem with entropy regularization is solved, and the transmission cost is used as the optimal transmission distance.

5. The multimodal defense method based on fast and slow thinking collaboration and online updates according to claim 1, characterized in that, The loss function includes a weighted combination of contrastive semantic loss and cross-modal fine-grained alignment loss; the contrastive semantic loss is calculated based on the global image features and global text features of the sanitized sample pair, and the cross-modal fine-grained alignment loss is calculated based on the optimal transmission distance between the image block feature set and the text lexical feature set of the sanitized sample pair.

6. The multimodal defense method based on fast and slow thinking collaboration and online updates according to claim 5, characterized in that, In the process of calculating the optimal transmission distance between the image block feature set and the text word feature set of the purified sample pair, the image block feature set of the purified sample pair extracted by the student model is used as the source distribution, and the text word feature set of the purified sample pair extracted by the student model is used as the target distribution. A similarity matrix is ​​constructed using the cosine similarity between the image block features and the text word features, and the complement of the similarity matrix is ​​used as the cost matrix. The optimal transmission problem with entropy regularization is solved to obtain the transmission cost as the optimal transmission distance.

7. The multimodal defense method based on fast and slow thinking collaboration and online updates according to claim 1, characterized in that, The process of synchronizing the updated parameters to the teacher model and the momentum text encoder includes: The updated student model parameters are updated using an exponential moving average, and the teacher model does not receive gradient backpropagation. The parameters of the momentum text encoder in the updated student model are updated using an exponential moving average.

8. The multimodal defense method based on fast and slow thinking collaboration and online updates according to claim 1, characterized in that, The candidate text queue contains a set of lexical features obtained by encoding the original text of the current suspicious sample pair with the momentum text encoder, and a set of lexical features of the text encoded with the momentum text encoder in the historical batches. The candidate text queue is maintained according to a first-in-first-out mechanism. After each training batch is completed, the set of lexical features obtained by encoding all texts in this batch with the momentum text encoder is appended to the end of the queue. If the queue capacity is full, the earliest entry of the same number of records is eliminated.

9. The multimodal defense method based on fast and slow thinking collaboration and online updates according to claim 1, characterized in that, The purified sample pairs processed by S2 and their priority scores are stored in the priority experience replay buffer. During each model update, historical samples are sampled from the buffer according to the priority scores and mixed with the current batch data for training.

10. A system for implementing the multimodal defense method based on fast and slow thinking collaboration and online updates as described in any one of claims 1 to 9, characterized in that, include: The Quick Thinking subsystem includes a teacher model, which is used to calculate the consistency confidence of text-image pairs arriving in streaming mode, and output a match decision, a non-match decision, or mark the text-image pair as a suspicious sample pair. The slow-thinking subsystem, which includes a student model, is used to process suspicious sample pairs marked by the fast-thinking subsystem into clean sample pairs by combining a candidate text queue maintained by the momentum text encoder. The online update subsystem is used to calculate the loss function using the cleaned sample pairs obtained from the student model and the slow thinking subsystem, update the parameters of the student model, and synchronize the updated parameters to the teacher model and the momentum text encoder.

Images