Multi-modal large model defense method and system based on visual text consistency constraint

Abstract

The application relates to the field of artificial intelligence security technology and discloses a multi-modal large model defense method and system based on visual-text consistency constraint, which has the technical scheme as follows: multi-modal feature extraction, visual-text consistency constraint calculation, self-adaptive threshold judgment, double-dimension consistency verification of fused semantic direction and statistical distribution characteristics, a self-adaptive weight distribution mechanism based on noise disturbance stability, and a cross-scene self-adaptive threshold judgment strategy, so that the method can accurately intercept picture-text collaborative attack, significantly improve the recognition accuracy of small disturbance samples, and reduce the cross-scene misjudgment rate.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and more specifically, to a multimodal large model defense method and system based on visual text consistency constraints. Background Technology

[0002] In recent years, although multimodal large models have been widely used in high-risk fields such as autonomous driving and content moderation, their adversarial attack defense systems still have obvious vulnerabilities: existing defense mechanisms mostly follow the single-modal independent detection approach and lack in-depth verification of cross-modal semantic associations between images and text, making it easy for collaborative attacks such as "fake images and real text" to bypass defenses; at the same time, traditional alignment detection based on models such as CLIP mainly relies on single indicators such as cosine similarity, which only measures the consistency of feature direction and ignores the slight disruption of feature statistical distribution by adversarial perturbations, resulting in insufficient sensitivity to small perturbation attacks; in addition, fixed threshold judgment strategies are difficult to adapt to the differences in data distribution in different business scenarios, and often face problems of high false positive rate and weak generalization ability in cross-scenario migration, failing to meet the security needs of complex and ever-changing scenarios.

[0003] Therefore, this invention provides a multimodal large model defense method and system based on visual text consistency constraints, which improves the above-mentioned technical problems. Summary of the Invention

[0004] This disclosure aims to address the shortcomings of existing technologies by providing a multimodal large model defense method and system based on visual text consistency constraints. The present invention solves the problem of cross-modal adversarial attack defense vulnerabilities faced by multimodal large models through the synergy of three major modules: multimodal feature extraction, dual-dimensional consistency constraints, and adaptive threshold determination.

[0005] To achieve the above objectives, the present disclosure proposes the following technical solutions:

[0006] In a first aspect, embodiments of this disclosure propose a multimodal large model defense method based on visual text consistency constraints, comprising the following steps:

[0007] S1. Multimodal feature extraction: Obtain the image data and text data to be detected, extract features using pre-trained visual encoders and text encoders respectively, and map the extracted features to a visual feature vector V and a text feature vector T of the same dimension;

[0008] S2. Visual-Text Consistency Constraint Calculation: The visual feature vector V and the text feature vector T are semantically interacted using a bidirectional cross-attention mechanism; based on the interacted features, the cosine similarity index, which measures semantic direction consistency, and the normalized mutual information index, which measures statistical distribution correlation, are calculated respectively; noise perturbation is applied to the features and adaptive weights are calculated based on the stability of each index under the perturbation, and the weighted fusion is used to obtain the final consistency score S.

[0009] S3. Adaptive Threshold Determination: Identify the business scenario to which the current data belongs and obtain the corresponding defense threshold for that scenario. If the consistency score S is less than the defense threshold If the data is flagged as an adversarial example, it will be blocked; otherwise, the data will be allowed into the multimodal large model.

[0010] As a preferred technical solution of the present invention, in step S1, the specific process of multimodal feature extraction includes: for image data, after denoising and enhancement preprocessing, inputting it into a ViT-Base architecture visual encoder, extracting classification token features, and obtaining the visual feature vector V through linear mapping and layer normalization; for text data, after word segmentation and length standardization processing, inputting it into a RoBERTa-Large architecture text encoder, and performing global average pooling and layer normalization on the output sequence to obtain the text feature vector T; the dimensions of the visual feature vector V and the text feature vector T are both 1024-dimensional.

[0011] As a preferred embodiment of the present invention, in step S2, the semantic interaction using a bidirectional cross-attention mechanism includes: calculating the attention features from visual to text respectively. Attention features from text to vision ;Will spliced ​​with the original visual feature V, The features are concatenated with the original text features T, and then compressed back to their original dimensions through bottleneck layers containing linear mapping layers and activation functions to obtain the interactive visual features. and post-interaction text features .

[0012] As a preferred embodiment of the present invention, in step S2, the calculation of the normalized mutual information index adopts an adaptive binning strategy, specifically including: calculating the interquartile range of the feature vector. Using the Freedman-Diaconis criterion, based on the interquartile range... The number of bins, K, is dynamically calculated based on the feature dimension N; the continuous feature vectors are discretized into a bin index sequence based on the number of bins, K; and the discretized post-interaction visual features are calculated. and post-interaction text features The mutual information between them is calculated, and symmetric normalization is performed using entropy to obtain a normalized mutual information index. .

[0013] As a preferred embodiment of the present invention, the formula for calculating the number of boxes K is as follows:

[0014] ;

[0015] Where F is the eigenvector, and max(F) and min(F) are the maximum and minimum values ​​of the eigenvector elements, respectively. This indicates rounding up to the nearest integer.

[0016] As a preferred embodiment of the present invention, in step S2, the adaptive weight calculation based on stability under perturbation specifically includes: adding Gaussian noise multiple times to the visual features and text features after interaction to generate multiple sets of perturbation features; and calculating the cosine similarity variance under the multiple sets of perturbation features. and normalized mutual information variance ; Calculate the confidence score for each indicator based on variance:

[0017] , ;

[0018] in, The sensitivity factor is preset; the fusion weight is calculated based on the confidence score.

[0019] ;

[0020] in, To prevent small positive numbers with a denominator of zero;

[0021] The final consistency score is calculated using the following formula:

[0022] ;

[0023] in, This is the normalized cosine similarity index.

[0024] As a preferred embodiment of the present invention, step S3 further includes a secondary verification mechanism: pre-setting a high consistency interval, a medium consistency interval, and a low consistency interval; if the consistency score S falls into the medium consistency interval, the current feature is frozen, a small amount of noise is added to the feature again, and the score after perturbation is calculated. ; Calculate the score difference ,like If the fluctuation exceeds the preset threshold, it is identified as an adversarial example and blocked.

[0025] As a preferred technical solution of the present invention, the defense threshold The threshold is determined through offline training, which includes: constructing a hybrid dataset containing semantic misalignment samples, gradient-based micro-perturbation samples, and core element replacement samples; traversing candidate thresholds on the validation set, calculating the defense misjudgment rate and accuracy under each candidate threshold; selecting the candidate threshold with the highest accuracy and a misjudgment rate not higher than a preset tolerance value as the optimal defense threshold for the scenario and storing it.

[0026] As a preferred technical solution of the present invention, in step S3, the defense threshold corresponding to the scenario is obtained. This includes: extracting the business interface ID or source system identifier from the metadata of the business request as a scene tag; using the scene tag to query the corresponding optimal threshold in a pre-stored database; if the query fails, using a preset default threshold.

[0027] Secondly, embodiments of this disclosure propose a multimodal large model defense system based on visual text consistency constraints, the system comprising:

[0028] The multimodal feature extraction module receives image and text data, preprocesses them, extracts features through a visual encoder and a text encoder respectively, aligns the dimensions, and outputs visual feature vectors and text feature vectors.

[0029] The visual-text consistency constraint module is used to interact visual and text features through a bidirectional cross-attention mechanism, calculate cosine similarity and normalized mutual information based on adaptive binning in parallel, and dynamically assign weights according to the variance of features under noise interference to calculate consistency scores.

[0030] The adaptive threshold determination module is used to identify business scenarios and query corresponding defense thresholds, compare the consistency score with the defense thresholds, and generate a pass or block instruction for multimodal large models based on the comparison result.

[0031] In summary, the present invention has the following beneficial effects:

[0032] Firstly, this invention constructs a deeply coupled image-text semantic consistency verification mechanism, effectively solving the problem that existing defense systems fragment multimodal data processing and struggle to identify cross-modal collaborative attacks. Through a bidirectional cross-attention mechanism, visual and textual features engage in deep semantic interaction. The system can simultaneously compare semantic relationships between modalities during the inference phase, thereby accurately identifying complex adversarial samples such as "fake image, real text" and "real image, fake text," which are extremely difficult for traditional independent detection models to identify. This significantly improves the coverage of the defense mechanism in real-world attack scenarios.

[0033] Secondly, this invention proposes a judgment strategy based on dual-dimensional similarity fusion and adaptive weight allocation, overcoming the limitation of existing technologies that rely solely on single indicators such as cosine similarity and are insensitive to minor perturbations. By introducing normalized mutual information based on adaptive binning as the second verification dimension, this invention can capture subtle changes in the statistical distribution of feature vectors. Combined with a dynamic weighting mechanism based on noise perturbation variance, it automatically reduces the weight of instability indicators. Compared to detection methods that directly use the CLIP model, this invention improves the recognition rate of adversarial examples based on minor pixel perturbations and enhances the robustness of the model.

[0034] Thirdly, this invention designs an adaptive threshold determination module, which solves the technical problems of high false positive rate and weak generalization ability of fixed threshold strategies in cross-scenario migration. By offline learning of the data distribution characteristics of different business scenarios (such as autonomous driving, medical imaging, etc.), the optimal defense threshold that meets the constraint of low false positive rate is dynamically determined, enabling the defense mechanism to flexibly adapt to a variety of business scenarios with huge data differences.

[0035] Fourth, this invention adopts a non-intrusive, pluggable architecture design, reducing engineering deployment costs and system modification risks. Consistency verification and threshold determination are embedded as independent modules in the multimodal large-model inference front end, eliminating the need to reconstruct or retrain the underlying structure of the large model and avoiding inference efficiency degradation due to model modifications. This pluggable design makes the defense mechanism easy to maintain and upgrade, ensuring the real-time performance of existing business systems at a relatively low computational cost. Attached Figure Description

[0036] Figure 1 A framework diagram of a multimodal large model defense system based on visual text consistency constraints provided in an embodiment of the present invention;

[0037] Figure 2 This is a flowchart of multimodal feature extraction provided in an embodiment of the present invention;

[0038] Figure 3 This is a flowchart of the adaptive threshold training process provided in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the internal structure of the visual-text consistency constraint module provided in an embodiment of the present invention. Detailed Implementation

[0040] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0043] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0044] This disclosure aims to address the shortcomings of existing defense mechanisms, such as the lack of cross-modal correlation verification, the insensitivity of single similarity indicators to minor adversarial perturbations, and the difficulty of adapting fixed threshold strategies to diverse business scenarios. Therefore, this disclosure proposes a multimodal large-model defense method and system based on visual-text consistency constraints. It employs a two-dimensional consistency verification that integrates semantic direction and statistical distribution features, an adaptive weight allocation mechanism based on noise perturbation stability, and a cross-scenario adaptive threshold determination strategy. This achieves the goals of accurately intercepting image-text collaborative adversarial attacks, significantly improving the accuracy of identifying samples with minor perturbations, and reducing the cross-scenario false positive rate.

[0045] Please refer to Figure 1 , Figure 1 The diagram illustrates the framework of a multimodal large-scale defense system based on visual-text consistency constraints according to an embodiment of this disclosure. It mainly consists of a multimodal feature extraction module, a visual-text consistency constraint module, and an adaptive threshold determination module.

[0046] S1, Multimodal Feature Extraction Module, this module as follows: Figure 2 As shown, it is responsible for converting the input heterogeneous data into aligned feature vectors.

[0047] S1.1 Visual data processing.

[0048] Data format validation and enhancement: For low-quality images commonly seen in telecommunications operator scenarios (such as those taken with old mobile phones or compressed multiple times), Gaussian filtering is first used to remove high-frequency noise, followed by Laplacian sharpening to enhance image edge details. For video stream data, a dynamic extraction strategy based on keyframe quality scoring is employed.

[0049] Spatial and pixel normalization: The image is uniformly scaled and gray background is filled at the edges to accommodate different aspect ratios. Then, the pixel values ​​are normalized to the range of [-1, 1].

[0050] Feature extraction and representation: The preprocessed image is input into a pre-trained visual encoder. This embodiment uses the ViT-Base model because it is more sensitive to fine-grained semantic changes. During deployment, attention heads that contribute less to the cross-modal alignment task are removed to shorten inference time. The [CLS] classification tokens output by the model are extracted as initial visual features.

[0051] Unified feature dimensions: The initial visual features are passed through a linear mapping layer to increase the dimension to 1024. Then, layer normalization is performed to obtain the final 1024-dimensional visual features V.

[0052] S1.2 Text data processing.

[0053] Text cleaning and word segmentation: Special characters are filtered, and homonyms commonly used in adversarial attacks are detected and replaced. The Jieba word segmentation tool is used for Chinese word segmentation to ensure stability in general domains.

[0054] Length standardization: The text length is uniformly limited or padded to 64 characters. Any shortfall is padded with padding tokens; for excessively long texts, a sliding window strategy is used to extract key segments.

[0055] Feature extraction and representation: [CLS] and [SEP] tokens are added before the text sequence and input into the RoberTa-Large text encoder. The model is pre-tuned on a customer service dialogue corpus.

[0056] Feature enhancement and dimensionality unification: Global average pooling is performed on the initial text feature matrix along the sequence dimension, aggregating it into a 1024-dimensional vector. After layer normalization, the final 1024-dimensional text feature T is obtained.

[0057] S2, Visual-Text Consistency Constraint Module, this module as follows Figure 3 As shown, this is the core decision unit of the defense mechanism. It calculates the semantic matching degree between visual feature V and text feature T, and finally outputs a quantitative score S from 0 to 1. The higher the score, the more consistent the image and text are.

[0058] S2.1 Cross-modal attention interaction layer: Deep semantic interaction is achieved by using a bidirectional cross-attention mechanism.

[0059] Visual-to-text attention: Visual features V are used as the query vector Q, and text features T are used as both the key vector K and the value vector V.

[0060]

[0061] Text-to-visual attention: Text features T are used as the query vector Q, and visual features V are used as both the key vector K and the value vector V.

[0062]

[0063] S2.2 Interactive Feature Fusion: Integrating Attention Output Features , The features are concatenated with the original features V and T respectively, resulting in a dimension of 2048. Then, a bottleneck layer (composed of a linear mapping and a ReLU activation function) is used to compress the feature dimension back to d=1024, which is denoted as the post-interaction visual feature. and text features .

[0064] S2.3, Dual-dimensional similarity fusion algorithm: In order to cope with small perturbation attacks, this embodiment calculates cosine similarity and normalized mutual information simultaneously.

[0065] S2.3.1, Cosine similarity calculation, measures the degree of alignment of feature vectors in the semantic direction after interaction, used to quickly capture directional deviations in the feature space: Let the visual features after bidirectional cross-attention interaction be... Text features are The feature dimension is The formula for calculating cosine similarity is:

[0066]

[0067] in, and They are and The i-th element.

[0068] To ensure range consistency, a linear transformation is performed: .

[0069] S2.3.2 Normalized Mutual Information Calculation (Introducing Adaptive Binning): The 1024-dimensional feature vector is regarded as a data distribution containing 1024 samples. The correlation between the two distributions is measured by calculating the mutual information between them.

[0070] S2.3.2.1 Calculate the number of bins K based on the Freedman-Diaconis criterion using the feature vector F (i.e. or For example:

[0071] ① Calculate the interquartile range Sort F in ascending order as follows: Calculate the position of the lower quartile: Location Position of the upper quartile: Location .

[0072] Numerical values ​​were calculated using linear interpolation.

[0073] ;

[0074] ;

[0075] ;

[0076] like , recorded as .

[0077] ② Calculate the width of each compartment: Where N=1024 is the number of samples in the feature vector F, i.e., the feature dimension.

[0078] ③ Calculate the initial K value and limit its range:

[0079]

[0080] in, This is the floor function.

[0081] To avoid the value of the number of boxes K being too large or too small, its range is set to [5, 30], and the formula for calculating K is adjusted as follows: .

[0082] S2.3.2.2 Feature discretization, obtaining the discretized features and For eigenvalues Its binning index is:

[0083]

[0084] in, It is an eigenvector ( or ), This represents the floor function.

[0085] S2.3.2.3 Calculate mutual information I, using the following formula:

[0086]

[0087] in, It is a joint probability distribution. and It is a marginal probability distribution:

[0088] , ,

[0089] in, Representing the eigenvector The number of elements that fall into the i-th bin. It means that both conditions are met. Falling into the i-th box and The number of paired elements falling into the j-th box, N=1024.

[0090] S2.3.2.4 Calculation of Symmetric Normalization The calculation formula is as follows:

[0091]

[0092] The entropy H is defined as follows:

[0093] , .

[0094] S2.3.3 Adaptive weight allocation mechanism: This mechanism utilizes the characteristic that "adversarial samples are sensitive to small perturbations" to determine the weights by calculating the variance of the index under noise.

[0095] S2.3.3.1, Generating perturbation characteristics: for and Add Gaussian noise with a mean of 0 and a standard deviation of 0.01, repeat 10 times, and generate 10 sets of perturbation features:

[0096]

[0097] S2.3.3.2 Calculation of local variance of the index:

[0098] Cosine similarity variance Calculate 10 sets of disturbance characteristics Find its variance.

[0099] Mutual information variance Set the set of sub-boxes 7 groups were traversed and calculated. Find the value and calculate its variance.

[0100] S2.3.3.3 Calculate the credibility score for each similarity index:

[0101] ,

[0102] in, =10 is the scaling factor. The smaller the variance, the closer the confidence score is to 1, indicating that the indicator is more stable.

[0103] Based on the credibility score from the previous step, the fusion weight is calculated using a linear normalization method. The higher the credibility, the higher the weight percentage.

[0104]

[0105] in, 1e-6 is a very small positive number, and this is to prevent errors when the denominator is zero.

[0106] S2.3.3.4 Calculate the eventual consistency score: .

[0107] S2.4 Result Judgment and Secondary Verification.

[0108] High consistency interval (S≥0.8): considered risk-free and passed to the threshold determination module.

[0109] Low consistency interval (S<0.5): identified as high-risk adversarial sample and directly blocked.

[0110] For the consistency interval (0.5≤S<0.8): perform a second check.

[0111] Secondary verification: In normal samples, the semantics of the image and text are consistent, and the consistency score changes little after adding minor noise. Adversarial samples only indicate semantic consistency; minor noise can disrupt this balance, leading to large score fluctuations. The specific steps of secondary verification are as follows:

[0112] Freeze current features , Adding Gaussian noise with a mean of 0 and a standard deviation of 0.01 again yields the perturbation characteristics:

[0113]

[0114] Calculate the consistency score after perturbation .

[0115] Calculate the difference .like If the sample is identified as an adversarial example (it takes advantage of the fact that adversarial examples are sensitive to small noises, which causes large fluctuations in scores); otherwise, it is identified as a normal sample.

[0116] S3, Adaptive Threshold Determination Module. This module is as follows: Figure 4 As shown, this is used to solve the problem of data distribution differences across scenarios.

[0117] S3.1, Offline threshold training.

[0118] Dataset Construction: Construct an adversarial sample set containing semantic misalignment (image and text mismatch), minor perturbations (using the PGD algorithm, setting the number of iterations to 10, step size to 0.01, and L2 norm ≤ 0.06), and core element replacements (such as replacing "red light" with "green light").

[0119] Traversal optimization: Traverse the candidate thresholds on the validation set (step size 0.01).

[0120] Determination principle: Prioritize screening candidate thresholds with a false positive rate (FP / (FP+TN)) ≤ 5%; among all candidate thresholds that meet this condition, select the threshold with the highest defense accuracy ((TP+TN) / Total) and determine it as the defense threshold for this scenario (denoted as ). ), and associate scene tags with The mapping relationship is stored in the database.

[0121] S3.2, Online threshold call.

[0122] Scene identification: Extract the business interface ID or source system identifier from the business request metadata. If missing, use a preset default defense threshold (e.g., ...). =0.7).

[0123] The judgment and response process involves querying the database for the defense threshold corresponding to the current scenario. The consistency score S is compared with... Comparison:

[0124] If S < If it is identified as an adversarial sample, an interception command is generated and logged.

[0125] If S≥ The sample was determined to be normal and the data was released.

[0126] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal large model defense method based on visual-text consistency constraints, characterized in that, The method includes the following steps: S1. Multimodal feature extraction: Obtain the image data and text data to be detected, extract features using pre-trained visual encoder and text encoder respectively, and map the extracted features to a visual feature vector V and a text feature vector T of the same dimension; S2. Visual-Text Consistency Constraint Calculation: A bidirectional cross-attention mechanism is used to semantically interact with the visual feature vector V and the text feature vector T; based on the interacted visual features... and text features The cosine similarity index, which measures semantic direction consistency, and the normalized mutual information index, which measures statistical distribution correlation, are calculated respectively. Noise perturbation is applied to the features and adaptive weights are calculated based on the stability of each index under the perturbation. The weighted fusion is then used to obtain the final consistency score S. Obtain the normalized mutual information index The process is as follows: calculate the discretized post-interaction visual features. and post-interaction text features Mutual information between them, and use entropy for symmetric normalization; S3. Adaptive Threshold Determination: Identify the business scenario to which the current data belongs and obtain the corresponding defense threshold for that scenario. If the consistency score S is less than the defense threshold If it is, it is identified as an adversarial example and blocked; otherwise, the data is allowed to enter the multimodal large model. In step S2, the adaptive weight calculation based on stability under perturbation specifically includes: adding Gaussian noise multiple times to the visual features and text features after interaction to generate multiple sets of perturbation features; and calculating the cosine similarity variance under the multiple sets of perturbation features. and normalized mutual information variance ; Calculate the confidence score for each indicator based on variance: ； in, The sensitivity factor is preset; adaptive weights are calculated based on the confidence score. ； in, To prevent small positive numbers with a denominator of zero; The final consistency score is calculated using the following formula: ； in, This is the normalized cosine similarity index.

2. The multimodal large model defense method based on visual-text consistency constraints according to claim 1, characterized in that, In step S1, the specific process of multimodal feature extraction includes: for image data, after denoising and enhancement preprocessing, the data is input into a ViT-Base architecture visual encoder to extract classification token features and obtain the visual feature vector V through linear mapping and layer normalization; for text data, after word segmentation and length standardization, the data is input into a RoBERTa-Large architecture text encoder to obtain the text feature vector T by performing global average pooling and layer normalization on the output sequence; both the visual feature vector V and the text feature vector T have a dimension of 1024.

3. The multimodal large model defense method based on visual-text consistency constraints according to claim 1, characterized in that, In step S2, the semantic interaction using the bidirectional cross-attention mechanism includes: calculating the attention features from visual to text respectively. Attention features from text to vision ;Will spliced ​​with the original visual feature V, The features are concatenated with the original text features T, and then compressed back to their original dimensions through bottleneck layers containing linear mapping layers and activation functions to obtain the interactive visual features. and post-interaction text features .

4. The multimodal large model defense method based on visual-text consistency constraints according to claim 1, characterized in that, In step S2, the calculation of the normalized mutual information index adopts an adaptive binning strategy, specifically including: calculating the interquartile range of the feature vectors. Using the Freedman-Diaconis criterion, based on the interquartile range... The number of bins K is dynamically calculated based on the feature dimension N; the continuous feature vectors are discretized into a bin index sequence based on the number of bins K.

5. The multimodal large model defense method based on visual text consistency constraints according to claim 4, characterized in that, The formula for calculating the number of boxes K is: ； Where F is the eigenvector, and max(F) and min(F) are the maximum and minimum values ​​of the eigenvector elements, respectively. This indicates rounding up to the nearest integer.

6. The multimodal large model defense method based on visual text consistency constraints according to claim 1, characterized in that, Step S3 also includes a secondary verification mechanism: pre-setting high consistency intervals, medium consistency intervals, and low consistency intervals; if the consistency score S falls into the medium consistency interval, the current feature is frozen, a small amount of noise is added to the feature again, and the perturbed score is calculated. ; Calculate the score difference ,like If the fluctuation exceeds the preset threshold, it is identified as an adversarial example and blocked.

7. The multimodal large model defense method based on visual-text consistency constraints according to claim 1, characterized in that, The defense threshold The threshold is determined through offline training, which includes: constructing a hybrid dataset containing semantic misalignment samples, gradient-based micro-perturbation samples, and core element replacement samples; traversing candidate thresholds on the validation set, calculating the defense misjudgment rate and accuracy under each candidate threshold; selecting the candidate threshold with the highest accuracy and a misjudgment rate not higher than a preset tolerance value as the optimal defense threshold for the scenario and storing it.

8. The multimodal large model defense method based on visual text consistency constraints according to claim 1, characterized in that, In step S3, the defense threshold corresponding to the scenario is obtained. This includes: extracting the business interface ID or source system identifier from the metadata of the business request as a scene tag; using the scene tag to query the corresponding optimal threshold in a pre-stored database; if the query fails, using a preset default threshold.

9. A multimodal large-scale model defense system based on visual-text consistency constraints, characterized in that: The system is used to implement the multimodal large model defense method based on visual text consistency constraints as described in any one of claims 1 to 8, and the system includes: The multimodal feature extraction module receives image and text data, preprocesses them, extracts features through a visual encoder and a text encoder respectively, aligns the dimensions, and outputs visual feature vectors and text feature vectors. The visual-text consistency constraint module is used to interact visual and text features through a bidirectional cross-attention mechanism, calculate cosine similarity and normalized mutual information based on adaptive binning in parallel, and dynamically assign weights according to the variance of features under noise interference to calculate consistency scores. The adaptive threshold determination module is used to identify business scenarios and query the corresponding defense thresholds, compare the consistency score with the defense thresholds, and generate a pass or block instruction for the multimodal large model based on the comparison result.

Images