Method for evaluating safety of multi-modal large model based on spatiotemporal collaborative gradient organization
Patent Information
- Application Number
- CN202611020109.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-09
- Publication Date
- 2026-10-09
AI Technical Summary
上述方法能够在一定程度上测试多模态大模型对扰动样本的敏感程度，但在损失反向传播获得图像侧梯度后，往往直接使用原始图像侧梯度进行样本更新
[0051]本发明在图文匹配安全性评估损失反向传播之后，对图像侧梯度进行时空协同梯度组织，通过时间维度上的跨迭代历史梯度累积降低单次梯度波动对扰动测试样本生成过程的影响，使生成的测试样本能够更稳定地反映多模态图文匹配关系中的共性脆弱方向；在空间维度上根据图像侧梯度的幅值响应生成残差空间重加权掩码，对不同空间位置的更新幅度进行轻量调制，同时保留全局更新路径，并与时间维度得到的跨迭代稳定更新方向协同形成扰动更新方向，减少过度依赖局部稀疏区域造成的测试偏差；能够生成满足扰动约束的扰动测试样本，并将其输入待评估多模态大模型量化模型安全性，从而提高异构多模态大模型在测试场景下的安全性评估有效性；适用于视觉语言预训练大模型、多模态检索大模型以及通用多模态大模型的安全性测试，可用于搜索推荐、内容理解、图文检索和交互式人工智能系统部署前的鲁棒性评估。
Smart Images
Figure CN122886701A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a method for security assessment of multimodal large models based on spatiotemporal collaborative gradient organization. Background Technology
[0002] With the development of deep learning, large-scale pre-training techniques, and multimodal large models, these models have been widely applied in scenarios such as image-text retrieval, content recommendation, image understanding, visual question answering, intelligent search, and interactive artificial intelligence systems. These models typically extract visual and textual features through visual encoders, text encoders, or cross-modal fusion modules, and calculate the matching relationship between images and text in a unified semantic space. For general-purpose multimodal large models capable of image-text input, their security assessment process in image-text matching tasks can be incorporated by configuring image-text matching scores, image-text retrieval ranking results, top-ranked text retrieval results, or top-ranked image retrieval results.
[0003] In image-text retrieval tasks, multimodal large model systems need to return text matching an image given a given image, or images matching text given a given text. If the model is sensitive to minor perturbations in the input image or text, problems such as disrupted image-text matching, changes in search ranking, and replacement of previously correct matches may occur. These issues reduce the reliability of multimodal large model systems in open environments; therefore, a security assessment is necessary before model deployment or during model updates.
[0004] Existing methods for assessing the security of large multimodal models typically begin with constructing a forward image-text matching loss. This loss is often built using methods such as similarity relationships between image and text features, semantic projection, input state sampling, data augmentation, or multi-scale transformations. Perturbation test samples are then generated based on this loss. While these methods can test the sensitivity of large multimodal models to perturbation samples to some extent, after obtaining the image-side gradient through backpropagation, they often directly use the original image-side gradient for sample updates.
[0005] Since the gradient of the original image may contain both the cross-model stabilizing direction and the local response of the source model, directly using this gradient can easily cause the perturbation test samples to become overly dependent on the local structural features of the source model. In heterogeneous scenarios where the source multimodal large model and the multimodal large model to be evaluated employ different visual encoders, different text encoders, different pre-training targets, or different cross-modal fusion mechanisms, the above issues can affect the effectiveness of security assessment. Summary of the Invention
[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal large model security assessment method based on spatiotemporal cooperative gradient organization. This method can cooperatively organize the image-side gradients obtained by backpropagation in both temporal and spatial dimensions, making them more suitable for cross-model perturbation test samples and ensuring the effectiveness of current multimodal large model security assessment.
[0007] To achieve the above objectives, the present invention is implemented using the following technical solution:
[0008] This invention provides a method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization, including:
[0009] Obtain an image text sample set;
[0010] The image and text sample set is input into a pre-constructed source multimodal large model, which outputs image features and text features. Based on the matching relationship between image features and text features, an image-text matching security assessment loss is constructed.
[0011] Backpropagation is performed on the image-text matching security assessment loss to obtain the image-side gradient;
[0012] In the time dimension, the image-side gradient is purified by time gradient purification to obtain a cross-iteration stable update direction. In the spatial dimension, a residual space reweighting mask is generated based on the magnitude response of the image-side gradient. Based on the cross-iteration stable update direction and the residual space reweighting mask, a perturbation update direction is generated collaboratively.
[0013] The image text sample set is updated according to the perturbation update direction to generate a perturbation test sample set;
[0014] The image and text sample set and the perturbation test sample set are sequentially input into the pre-constructed multimodal large model to be evaluated to obtain the changes in image and text retrieval results before and after perturbation;
[0015] Based on the changes in image and text retrieval results before and after the perturbation, a security assessment result for the multimodal large model to be evaluated in the image and text matching task is generated.
[0016] Optionally, the image-text sample set is input into a pre-constructed source multimodal large model, which outputs image features and text features. Based on the matching relationship between image features and text features, an image-text matching security assessment loss is constructed, including:
[0017] The image samples from the image-text sample set are input into the image encoder in the source multimodal large model, and the output image features are generated.
[0018] The text description samples from the image text sample set are input into the text encoder in the source multimodal large model to output text features;
[0019] The image-text matching security assessment loss is calculated based on at least one of the following methods: similarity relationship between image features and text features, semantic projection relationship, input state sampling relationship, multi-scale transformation relationship, and text-side perturbation test relationship.
[0020] Optionally, the image-text matching security assessment loss is expressed as:
[0021] ;
[0022] in, This indicates the loss in the security assessment of image-text matching; This represents the semantic projection image-text matching loss function; This represents the input variable of the semantic projection graph-text matching loss function; Indicates the first The image test samples obtained in the nth iteration, where... This indicates that the image test sample is in a perturbation test state. Indicates the image side, ; This represents a text description sample; Represents the semantic projection matrix or image-text matching constraint parameters; Represents the image feature extraction function of a large multimodal model; This represents the text feature extraction function for a large multimodal model.
[0023] Optionally, the image-side gradient is purified temporally to obtain a stable update direction across iterations, including:
[0024] The image-side gradient of the current iteration is normalized to obtain the normalized image-side gradient of the current iteration;
[0025] Based on the historical gradient decay coefficient, the normalized image gradient of the current iteration is accumulated across iterations with the historical accumulation direction of the previous iteration to obtain the stable update direction across iterations.
[0026] Optionally, the cross-iteration stable update direction is represented as:
[0027] ;
[0028] in, Indicates the first The stable update direction across iterations in the next iteration; Indicates the historical gradient decay coefficient; Indicates the first The historical cumulative direction of each iteration; Indicates the first The normalized image side gradient of the next iteration.
[0029] Optionally, a residual spatial reweighted mask is generated based on the magnitude response of the image-side gradient in the spatial dimension, including:
[0030] The absolute value of the image-side gradient is taken, and the absolute values of the image-side gradients are converged in the channel dimension to obtain a two-dimensional gradient magnitude response map.
[0031] Spatial pooling is performed on the two-dimensional gradient magnitude response map to obtain a region response map, and the response threshold in the region response map is determined according to a preset ratio.
[0032] Generate a spatial soft mask based on the regional response map and response threshold;
[0033] The spatial soft mask is upsampled to the same spatial size as the image sample. Based on the residual reweighting coefficient, the upsampled spatial soft mask is fused with the preset global preserved path to obtain the residual spatial reweighted mask.
[0034] Optionally, the spatial soft mask is represented as:
[0035] ;
[0036] in, Indicates the first The spatial soft mask for the next iteration; Indicates the first The region response map of the next iteration; Indicates the first The response threshold for the next iteration; Indicates the temperature coefficient; Represents the Sigmoid function;
[0037] The residual space reweighting mask is represented as follows:
[0038] ;
[0039] in, Indicates the first The residual space reweighting mask for the next iteration; Indicates the residual reweighting coefficient; Indicates the default globally reserved path; This indicates an upsampling operation.
[0040] Optionally, based on the cross-iteration stable update direction and the residual space reweighting mask, a perturbation update direction is collaboratively generated, including:
[0041] The sign update direction for the current iteration is determined based on the stable update direction across iterations;
[0042] The perturbation update direction of the current iteration is obtained by spatially modulating the symbol update direction of the current iteration based on the residual spatial reweighting mask;
[0043] The image text sample set is updated according to the perturbation update direction to generate a perturbation test sample set, including:
[0044] Based on the current iteration's perturbation update direction and update step size, update the image samples in the image text sample set, project the updated image samples back into the preset perturbation constraint range, obtain the image samples for the next iteration, and use them as the perturbation test samples for the next iteration.
[0045] Repeatedly iterate and update until the preset number of iterations is reached to obtain perturbation test samples. Based on the perturbation test samples and their corresponding text description samples, a perturbation test sample set is generated.
[0046] Optionally, the disturbance test sample is represented as:
[0047] ;
[0048] in, Indicates the first The perturbation test sample for the next iteration; Indicates the first Image samples from the next iteration; Indicates the update step size; Indicates the first The perturbation update direction in the next iteration; This represents the range of infinite norm perturbation constraints centered on the image sample; Indicates a projection operation; Represents image samples; This indicates the disturbance constraint radius or the maximum disturbance budget.
[0049] Optionally, the security assessment results of the multimodal large model to be evaluated in the image-text matching task include the first-order mismatch rate for text retrieval and the first-order mismatch rate for image retrieval.
[0050] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:
[0051] This invention, after backpropagation of the loss in image-text matching security assessment, performs spatiotemporal co-organized gradient organization on the image-side gradient. By accumulating historical gradients across iterations in the time dimension, it reduces the impact of single gradient fluctuations on the generation process of perturbation test samples, enabling the generated test samples to more stably reflect the common vulnerable directions in multimodal image-text matching relationships. In the spatial dimension, it generates a residual spatial reweighted mask based on the magnitude response of the image-side gradient, lightly modulating the update amplitude at different spatial locations while preserving the global update path. This mask, along with the stable update direction obtained across iterations in the time dimension, forms a perturbation update direction, reducing test bias caused by over-reliance on local sparse regions. It can generate perturbation test samples that meet perturbation constraints and input them into the multimodal large-scale model to be evaluated to quantify model security, thereby improving the effectiveness of security assessment of heterogeneous multimodal large-scale models in test scenarios. It is suitable for security testing of visual language pre-trained large-scale models, multimodal retrieval large-scale models, and general multimodal large-scale models, and can be used for robustness assessment before deployment in search recommendation, content understanding, image-text retrieval, and interactive artificial intelligence systems. Attached Figure Description
[0052] Figure 1 This is a flowchart illustrating the multimodal large model security assessment method based on spatiotemporal cooperative gradient organization provided in an embodiment of the present invention.
[0053] Figure 2 A schematic diagram illustrating the process of generating perturbation update direction for spatiotemporal cooperative gradient organization provided in an embodiment of the present invention;
[0054] Figure 3 This is a schematic diagram of the process for generating a residual space reweighting mask provided in an embodiment of the present invention;
[0055] Figure 4 This is a schematic diagram of the process for generating the security assessment results of a large multimodal model to be evaluated in a text-image matching task, as provided in an embodiment of the present invention. Detailed Implementation
[0056] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0057] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0058] Example 1
[0059] This embodiment introduces a method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization, including:
[0060] Obtain an image text sample set;
[0061] The image and text sample set is input into a pre-constructed source multimodal large model, which outputs image features and text features. Based on the matching relationship between image features and text features, an image-text matching security assessment loss is constructed.
[0062] Backpropagation is performed on the image-text matching security assessment loss to obtain the image-side gradient;
[0063] In the time dimension, the image-side gradient is purified by time gradient purification to obtain a cross-iteration stable update direction. In the spatial dimension, a residual space reweighting mask is generated based on the magnitude response of the image-side gradient. Based on the cross-iteration stable update direction and the residual space reweighting mask, a perturbation update direction is generated collaboratively.
[0064] The image text sample set is updated according to the perturbation update direction to generate a perturbation test sample set;
[0065] The image and text sample set and the perturbation test sample set are sequentially input into the pre-constructed multimodal large model to be evaluated to obtain the changes in image and text retrieval results before and after perturbation;
[0066] Based on the changes in image and text retrieval results before and after the perturbation, a security assessment result for the multimodal large model to be evaluated in the image and text matching task is generated.
[0067] In this embodiment, after constructing the image-text matching security assessment loss, the image-side gradient obtained by backpropagation is co-organized in terms of time and space dimensions to make it more suitable for generating cross-model perturbation test samples. Based on this, the security and robustness of the multimodal large model to be evaluated in heterogeneous scenarios are assessed, which is a technical problem that needs to be solved in the current security assessment of multimodal large models.
[0068] Example 2
[0069] Based on Example 1, such as Figure 1 As shown in the figure, this embodiment introduces a method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization, including the following steps:
[0070] Step 1: Construct the image-text matching security assessment loss, specifically as follows:
[0071] Obtain an image-text sample set, which includes image samples and text description samples.
[0072] The input data is an image and text sample set. The model used in this embodiment includes a source multimodal large model and a multimodal large model to be evaluated. The source multimodal large model is used to generate perturbation test samples, and the multimodal large model to be evaluated is used to receive perturbation test samples and output image and text retrieval results.
[0073] The source multimodal large model and the multimodal large model to be evaluated can adopt the same structure, or they can adopt different visual encoders, different text encoders, different pre-training targets or different cross-modal fusion mechanisms; and they can adopt a general multimodal large model that can be configured to output image-text matching scores, image-text retrieval ranking results, text retrieval first results or image retrieval first results. To illustrate the security assessment effect in the image-text matching task, in this embodiment, visual language pre-training models or multimodal retrieval models such as the Align Before Fuse (ALBEF) model, the Triple Contrastive Learning (TCL) model, the Contrastive Language-Image Pre-training with VisionTransformer Image Encoder (CLIPViT) model, and the Contrastive Language-Image Pre-training with Convolutional Neural Network Image Encoder (CLIPCNN) model can also be used as the source multimodal model and the multimodal model to be evaluated.
[0074] The image-text sample set was obtained by preprocessing the image-text matching dataset, namely:
[0075] The image samples in the image-text matching dataset are resized, pixel normalized, and batch processed to obtain image samples in the image-text sample set.
[0076] The text description samples in the image-text matching dataset are segmented, encoded, and batch processed to obtain the text description samples in the image-text sample set.
[0077] The processed image samples and text description samples are combined to form an image-text sample set.
[0078] Input the image and text sample set into a pre-constructed source multimodal large model, and output image features and text features:
[0079] Input image samples from the image-text sample set into the image encoder in the source multimodal large model, and output image features;
[0080] The text description samples from the image text sample set are input into the text encoder in the source multimodal large model, and the text features are output.
[0081] Based on the matching relationship between image features and text features, a loss function for image-text matching security assessment is constructed:
[0082] The image-text matching security assessment loss is calculated based on at least one of the following methods: similarity relationship between image features and text features, semantic projection relationship, input state sampling relationship, multi-scale transformation relationship, and text-side perturbation test relationship. Represented as:
[0083] ;
[0084] in, This represents the semantic projection image-text matching loss function; This represents the input variable of the semantic projection graph-text matching loss function; Indicates the first The image test samples obtained in the nth iteration, where... This indicates that the image test sample is in a perturbation test state. Indicates the image side, Indicates the number of iterations; This represents a text description sample; Represents the semantic projection matrix or image-text matching constraint parameters; Represents the image feature extraction function of a large multimodal model; This represents the text feature extraction function for a large multimodal model.
[0085] This embodiment is not limited to a certain type of image-text matching security assessment loss, but rather performs spatiotemporal cooperative gradient organization on the image-side gradient obtained by backpropagation after obtaining the image-text matching security assessment loss.
[0086] Step 2: Backpropagation to obtain the image-side gradient, specifically:
[0087] Backpropagation is performed on the image-text matching security assessment loss to obtain the image-side gradient, i.e.:
[0088] The image sample is initialized as the current image test sample, and the current image test sample is set to a differentiable state. The image-text matching security evaluation loss is calculated on the current image test sample. The image-text matching security evaluation loss is backpropagated relative to the current image test sample to obtain the image-side gradient of the current iteration. The image sample initialization step is omitted in this embodiment.
[0089] The gradient of the image-text matching security assessment loss with respect to the current image test sample is obtained, which represents the direction of influence of each pixel position in the current image test sample on the image-text matching security assessment loss.
[0090] Step 3: Spatiotemporal cooperative gradient organization generates perturbation update directions, specifically as follows:
[0091] like Figure 2 As shown, temporal gradient purification is performed on the image-side gradient in the time dimension to obtain a stable update direction across iterations. In the spatial dimension, a residual space reweighting mask is generated based on the magnitude response of the image-side gradient. Based on the stable update direction across iterations and the residual space reweighting mask, a perturbation update direction is generated collaboratively.
[0092] Temporal gradient purification is a processing step in the temporal dimension of spatiotemporal co-organized gradients, used to extract persistent and stable directions from image side gradients over multiple iterations.
[0093] The image-side gradient of the current iteration is normalized to obtain the normalized image-side gradient of the current iteration.
[0094] Based on the historical gradient decay coefficient, the normalized image gradient of the current iteration is accumulated across iterations with the historical accumulation direction of the previous iteration to obtain the stable update direction across iterations. The cross-iteration stable update direction of the next iteration Represented as:
[0095] ;
[0096] in, Indicates the historical gradient decay coefficient; Indicates the first The historical cumulative direction of each iteration; Indicates the first Normalized image side gradient of the next iteration.
[0097] Specifically, such as Figure 2 As shown, in the first In the current iteration, the image-side gradient is first normalized to obtain the normalized image-side gradient of the current iteration; then, the normalized image-side gradient of the current iteration is weighted and accumulated with the historical accumulation direction of the previous iteration to obtain the th iteration. The stable update direction across iterations in the next iteration.
[0098] The above processing can reduce the impact of gradient fluctuations during a single backpropagation on the generation process of perturbation test samples, and allow the perturbation update direction to retain more stable information that continues to appear across iterations.
[0099] Spatial gradient reweighting is a spatial dimension processing step of spatiotemporal co-gravity gradient organization, used to perform residual modulation on the perturbation update amplitude based on the amplitude response of the image side gradient at different spatial locations.
[0100] The absolute value of the image-side gradient is taken, and the absolute values of the image-side gradients are converged along the channel dimension to obtain a two-dimensional gradient magnitude response map.
[0101] Spatial pooling is performed on the two-dimensional gradient magnitude response map to obtain the regional response map, and the response threshold in the regional response map is determined according to a preset ratio.
[0102] Based on the regional response map and response threshold, generate a spatial soft mask; The spatial soft mask of the next iteration Represented as:
[0103] ;
[0104] in, Indicates the first The region response map of the next iteration; Indicates the first The response threshold for the next iteration; Indicates the temperature coefficient; This represents the Sigmoid function.
[0105] The spatial soft mask is upsampled to the same spatial size as the image sample. Based on the residual reweighting coefficients, the upsampled spatial soft mask is fused with the preset globally preserved path to obtain the residual spatial reweighted mask. Residual space reweighting mask for the next iteration Represented as:
[0106] ;
[0107] in, Indicates the residual reweighting coefficient; Indicates the default globally reserved path; This indicates an upsampling operation.
[0108] Specifically, such as Figure 3As shown, the gradient of the current image is converged along the channel dimension to obtain a two-dimensional gradient magnitude response map; spatial pooling is performed on the two-dimensional gradient magnitude response map to obtain a region response map; a response threshold is determined from the region response map according to a preset ratio; and a spatial soft mask is generated based on the region response map and the response threshold. The method for generating the spatial soft mask includes:
[0109] The absolute value of the gradient on the image side is taken and converged in the channel dimension to obtain a two-dimensional gradient magnitude response map; the two-dimensional gradient magnitude response map is spatially pooled to obtain a region response map; the response threshold in the region response map is determined according to a preset ratio; a spatial soft mask is generated based on the region response map, the response threshold, and the temperature coefficient.
[0110] Then, the spatial soft mask is upsampled to the same spatial size as the current image test sample and fused with the globally preserved path to obtain the residual spatial reweighted mask. Methods for generating the residual spatial reweighted mask include:
[0111] Upsample the spatial soft mask to the spatial size of the current image test sample; set the global preserved path; and fuse the upsampled spatial soft mask with the global preserved path according to the residual reweighting coefficient to obtain the residual spatial reweighted mask.
[0112] In the above process, a spatial soft mask is used to modulate the update amplitude at different locations based on the spatial response intensity of the image-side gradient; the globally preserved path is used to avoid relying solely on local high-response regions for updates, thereby preserving the overall perturbation test intensity. Thus, the spatial dimension residual modulation result can work synergistically with the cross-iteration stable update direction obtained in the temporal dimension to jointly form the perturbation update direction used to generate perturbation test samples.
[0113] Based on the stable update direction across iterations and the residual space reweighting mask, a perturbation update direction is jointly generated, namely:
[0114] The symbol update direction for the current iteration is determined based on the stable update direction across iterations; the symbol update direction for the current iteration is spatially modulated based on the residual spatial reweighting mask to obtain the perturbation update direction for the current iteration.
[0115] Step 4: Generate disturbance test samples, specifically:
[0116] The image text sample set is updated according to the perturbation update direction to generate the perturbation test sample set, i.e.:
[0117] Based on the current iteration's perturbation update direction and update step size, the image samples in the image-text sample set are updated. The updated image samples are then projected back into the preset perturbation constraint range to obtain the image samples for the next iteration, which are then used as the perturbation test samples for the next iteration. The perturbation test samples are represented as follows:
[0118] ;
[0119] in, Indicates the first The perturbation test sample for the next iteration; Indicates the update step size; Indicates the first The perturbation update direction in the next iteration; This represents the range of infinite norm perturbation constraints centered on the image sample; Indicates a projection operation; Represents image samples; This indicates the disturbance constraint radius or the maximum disturbance budget.
[0120] Repeat the iteration process, including loss calculation, backpropagation, temporal gradient purification, residual space reweighting, and sample update, until the preset number of iterations is reached to obtain perturbation test samples. Based on the perturbation test samples and their corresponding text description samples, generate a perturbation test sample set.
[0121] Step 5: The security assessment results of the multimodal large model to be evaluated in the image-text matching task are as follows:
[0122] The image and text sample set and the perturbation test sample set are sequentially input into the pre-constructed multimodal large model to be evaluated to obtain the changes in image and text retrieval results before and after perturbation.
[0123] Specifically, such as Figure 4 As shown, perturbation test samples generated by the source multimodal large model are input into one or more multimodal large models to be evaluated, namely, multimodal large model A, multimodal large model B, and multimodal large model C. The multimodal large models to be evaluated perform image-text matching calculations on the perturbation test samples and their corresponding text description samples, and output text retrieval results and image retrieval results.
[0124] Input the image samples into the multimodal large model to be evaluated to obtain the original image and text retrieval results.
[0125] Input the perturbation test samples into the multimodal large model to be evaluated to obtain the perturbation-enhanced image and text retrieval results.
[0126] By comparing the original image and text search results with the perturbation-treated image and text search results, the changes in the image and text search results can be obtained.
[0127] Based on the changes in image-text retrieval results before and after the perturbation, a security assessment result for the multimodal large model to be evaluated in the image-text matching task is generated. The security assessment result for the multimodal large model to be evaluated in the image-text matching task includes the first-order mismatch rate for text retrieval and the first-order mismatch rate for image retrieval.
[0128] Specifically, the image and text retrieval results before and after the input of the perturbation test sample are compared, and the first-order mismatch rate of text retrieval and the first-order mismatch rate of image retrieval are statistically analyzed as the security evaluation results of the multimodal large model to be evaluated in the image and text matching task.
[0129] The first-order mismatch rate in text retrieval measures the proportion of text retrieval results that are incorrect when the first text is retrieved after perturbation of the test sample input. The first-order mismatch rate in image retrieval measures the proportion of image retrieval results that are incorrect when the first image is retrieved after perturbation of the test sample input.
[0130] The higher the above indicators, the lower the security and robustness of the multimodal large model to be evaluated in the corresponding image-text matching task; the lower the above indicators, the higher the security and robustness of the multimodal large model to be evaluated in the corresponding image-text matching task.
[0131] In this embodiment, the Flickr30K image-text retrieval dataset (30,000 images) and the Microsoft Common Objects inContext (MSCOCO) dataset from the Flickr image sharing platform are used as the image-text sample set. The ALBEF, TCL, CLIPViT, and CLIPCNN models are used as the multimodal model set for experimental verification. This model combination covers differences in visual encoding structure and pre-training objectives, and can be used to illustrate the applicability of this invention in heterogeneous multimodal models and multimodal large-scale model security assessment scenarios. This experimental verification is only used to illustrate the feasibility and effectiveness of this invention in image-text matching security assessment tasks, and is not intended to limit the types of multimodal large-scale models to which this invention is applicable.
[0132] Example 3
[0133] Based on Example 1 or 2, this example introduces an experimental example of a multimodal large model security assessment method based on spatiotemporal cooperative gradient organization:
[0134] The perturbation constraint radius is set to 8 / 255, the number of iterations is set to 10, the update step size is set to 2 / 255, the historical gradient decay coefficient in temporal gradient purification is set to 1.0, the preset ratio in residual space reweighting is set to 80%, the temperature coefficient is set to 0.5, and the residual reweighting coefficient is set to 0.1. The above parameters are only used to illustrate one possible implementation; those skilled in the art can adjust them according to image resolution, model structure, and evaluation task. Relevant parameters are shown in Table 1.
[0135] Table 1 Relevant Parameters
[0136]
[0137] The first-order mismatch rate for text retrieval measures the proportion of image samples that originally matched text correctly, but whose first-order text retrieval result is incorrect after perturbing the test sample input. The first-order mismatch rate for image retrieval measures the proportion of text samples that originally matched images correctly, but whose first-order image retrieval result is incorrect after perturbing the test sample input. Higher values of these metrics indicate lower security and robustness of the multimodal large-scale model under evaluation in the corresponding image-text matching task; lower values indicate higher security and robustness of the multimodal large-scale model under evaluation in the corresponding image-text matching task. Security evaluation metrics are shown in Table 2.
[0138] Table 2 Safety Assessment Indicators
[0139]
[0140] The Flickr30K image-text retrieval dataset and the MSCOCO image-text retrieval dataset were used as the image-text sample set. The ALBEF model, TCL model, CLIPViT model, and CLIPCNN model were used as the multimodal model set for experimental verification. These model combinations cover differences in visual coding structure and pre-training objectives, and can be used to illustrate the applicability of this invention in heterogeneous multimodal models and large multimodal model security assessment scenarios. This experimental verification is only used to illustrate the feasibility and effectiveness of this invention in image-text matching security assessment tasks, and is not intended to limit the types of large multimodal models to which this invention is applicable. The experimental results are shown in Table 3.
[0141] Table 3 Experimental Results
[0142]
[0143] The experimental results show that by stabilizing the image side gradient in the temporal dimension and reweighting the spatial residuals through spatiotemporal collaborative gradient organization, perturbation test samples that are more suitable for cross-model evaluation can be generated, thereby improving the ability to detect robustness defects in the security evaluation scenarios of heterogeneous multimodal models and large multimodal models.
[0144] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0145] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0148] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.
Claims
1. A method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization, characterized in that, include: Obtain an image text sample set; The image and text sample set is input into a pre-constructed source multimodal large model, which outputs image features and text features. Based on the matching relationship between image features and text features, an image-text matching security assessment loss is constructed. Backpropagation is performed on the image-text matching security assessment loss to obtain the image-side gradient; In the time dimension, the image-side gradient is purified by time gradient purification to obtain a cross-iteration stable update direction. In the spatial dimension, a residual space reweighting mask is generated based on the magnitude response of the image-side gradient. Based on the cross-iteration stable update direction and the residual space reweighting mask, a perturbation update direction is generated collaboratively. The image text sample set is updated according to the perturbation update direction to generate a perturbation test sample set; The image and text sample set and the perturbation test sample set are sequentially input into the pre-constructed multimodal large model to be evaluated to obtain the changes in image and text retrieval results before and after perturbation; Based on the changes in image and text retrieval results before and after the perturbation, a security assessment result for the multimodal large model to be evaluated in the image and text matching task is generated.
2. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 1, characterized in that, The image-text sample set is input into a pre-constructed source multimodal large model, which outputs image features and text features. Based on the matching relationship between image features and text features, an image-text matching security assessment loss is constructed, including: The image samples from the image-text sample set are input into the image encoder in the source multimodal large model, and the output image features are generated. The text description samples from the image text sample set are input into the text encoder in the source multimodal large model to output text features; The image-text matching security assessment loss is calculated based on at least one of the following methods: similarity relationship between image features and text features, semantic projection relationship, input state sampling relationship, multi-scale transformation relationship, and text-side perturbation test relationship.
3. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 1 or 2, characterized in that, The security assessment loss for image-text matching is expressed as: ； in, This indicates the loss in the security assessment of image-text matching; This represents the semantic projection image-text matching loss function; This represents the input variable of the semantic projection graph-text matching loss function; Indicates the first The image test samples obtained in the nth iteration, where... This indicates that the image test sample is in a perturbation test state. Indicates the image side, ; This represents a text description sample; Represents the semantic projection matrix or image-text matching constraint parameters; Represents the image feature extraction function of a large multimodal model; This represents the text feature extraction function for a large multimodal model.
4. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 1, characterized in that, Temporal gradient purification is performed on the image-side gradient in the time dimension to obtain a stable update direction across iterations, including: The image-side gradient of the current iteration is normalized to obtain the normalized image-side gradient of the current iteration; Based on the historical gradient decay coefficient, the normalized image gradient of the current iteration is accumulated across iterations with the historical accumulation direction of the previous iteration to obtain the stable update direction across iterations.
5. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 1 or 4, characterized in that, The cross-iteration stable update direction is represented as follows: ； in, Indicates the first The stable update direction across iterations in the next iteration; Indicates the historical gradient decay coefficient; Indicates the first The historical cumulative direction of each iteration; Indicates the first The normalized image side gradient of the next iteration.
6. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 1, characterized in that, Based on the magnitude response of the image-side gradient in the spatial dimension, a residual spatial reweighted mask is generated, including: The absolute value of the image-side gradient is taken, and the absolute values of the image-side gradients are converged in the channel dimension to obtain a two-dimensional gradient magnitude response map. Spatial pooling is performed on the two-dimensional gradient magnitude response map to obtain a region response map, and the response threshold in the region response map is determined according to a preset ratio. Generate a spatial soft mask based on the regional response map and response threshold; The spatial soft mask is upsampled to the same spatial size as the image sample. Based on the residual reweighting coefficient, the upsampled spatial soft mask is fused with the preset global preserved path to obtain the residual spatial reweighted mask.
7. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 6, characterized in that, The spatial soft mask is represented as follows: ； in, Indicates the first The spatial soft mask for the next iteration; Indicates the first The region response map of the next iteration; Indicates the first The response threshold for the next iteration; Indicates the temperature coefficient; Represents the Sigmoid function; The residual space reweighting mask is represented as follows: ； in, Indicates the first The residual space reweighting mask for the next iteration; Indicates the residual reweighting coefficient; Indicates the default globally reserved path; This indicates an upsampling operation.
8. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 1, characterized in that, Based on the cross-iteration stable update direction and the residual space reweighting mask, a perturbation update direction is collaboratively generated, including: The sign update direction for the current iteration is determined based on the stable update direction across iterations; The perturbation update direction of the current iteration is obtained by spatially modulating the symbol update direction of the current iteration based on the residual spatial reweighting mask; The image text sample set is updated according to the perturbation update direction to generate a perturbation test sample set, including: Based on the current iteration's perturbation update direction and update step size, update the image samples in the image-text sample set, project the updated image samples back into the preset perturbation constraint range, obtain the image samples for the next iteration, and use them as the perturbation test samples for the next iteration. Repeatedly iterate and update until the preset number of iterations is reached to obtain perturbation test samples. Based on the perturbation test samples and their corresponding text description samples, a perturbation test sample set is generated.
9. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 8, characterized in that, The disturbance test sample is represented as follows: ； in, Indicates the first The perturbation test sample for the next iteration; Indicates the first Image samples from the next iteration; Indicates the update step size; Indicates the first The perturbation update direction in the next iteration; This represents the range of infinite norm perturbation constraints centered on the image sample; Indicates a projection operation; Represents image samples; This indicates the disturbance constraint radius or the maximum disturbance budget.
10. The method for security assessment of multimodal large models based on spatiotemporal cooperative gradient organization according to claim 1, characterized in that, The security assessment results of the multimodal large model to be evaluated in the image-text matching task include the first-order mismatch rate for text retrieval and the first-order mismatch rate for image retrieval.