Time sequence adversarial sample security evaluation method and system for multi-modal large model

Abstract

The application relates to the field of information technology, in particular to a time sequence adversarial sample safety evaluation method and system for a multimodal large model. The method comprises the following steps: acquiring data modalities of a to-be-evaluated multimodal large model, and constructing a downstream task; defining a time sequence consistency loss, defining a loss function for making the multimodal large model output an incorrect result, and generating a target function; adding adversarial perturbations on frames of original time sequence multimodal data; generating time sequence adversarial samples according to the optimized adversarial perturbations and the original time sequence multimodal data, and calculating to obtain perturbation imperceptibility; obtaining an incorrect output proportion according to the output of the to-be-evaluated multimodal large model under multiple time sequence adversarial samples; obtaining a time sequence robustness index according to the internal state of the to-be-evaluated multimodal large model under the multiple time sequence adversarial samples; and generating an evaluation result according to the incorrect output proportion, the time sequence robustness index and the perturbation imperceptibility.

Description

Technical Field

[0001] This application relates to the field of information technology, specifically to a method and system for evaluating the security of temporal adversarial samples for multimodal large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large language models (MLLMs) have become a research hotspot. They can fuse and process information from multiple modalities such as text, images, videos, and audio, and demonstrate powerful capabilities in complex tasks. However, the security issues of MLLMs are becoming increasingly prominent, especially their vulnerability to adversarial samples.

[0003] Currently, adversarial example attacks and security assessment methods for MLLM mainly focus on static modalities, such as images and text. For image modalities, attacks typically mislead the model by adding tiny, imperceptible perturbations to a single frame of the image; while for text modalities, attacks are achieved through synonym substitution or sentence reconstruction.

[0004] For modalities involving temporal information, such as video and audio, existing static modality evaluation methods have the following problems:

[0005] 1. Lack of temporal consistency: Simply applying adversarial attack methods for static images to every frame of a video results in a lack of coherence in the perturbations between frames. This temporal inconsistency is not only easily detected or filtered in practical applications, but also does not match real-world temporal attack scenarios, making it impossible to accurately assess the model's robustness under real-world temporal attacks.

[0006] 2. Incomplete Evaluation Metrics: Existing evaluation metrics primarily focus on attack success rate and perturbation magnitude, but fail to introduce robustness metrics specifically for time-series data. For example, key security characteristics such as the model's predictive stability over time series and the temporal diffusion effect of adversarial perturbations have not been quantitatively evaluated. This limits our comprehensive understanding of the model in dynamic environments.

[0007] 3. Low efficiency in generating temporal attack sequences: Optimizing adversarial perturbations for the entire video or audio sequence is not only computationally complex, but also makes it difficult to ensure that the perturbation maintains temporal continuity while efficiently achieving the attack target. Existing methods often neglect how to generate effective temporal adversarial examples with limited computational resources.

[0008] Therefore, in order to more comprehensively evaluate the security of MLLM in the temporal modality, it is urgent to study new techniques for generating and evaluating adversarial examples. Summary of the Invention

[0009] This specification describes a method and system for evaluating the security of temporal adversarial examples for multimodal large models through several embodiments.

[0010] Firstly, the embodiments of this specification provide a method for evaluating the security of temporal adversarial examples for multimodal large models, including the following steps:

[0011] Obtain the data modalities of the multimodal large model to be evaluated, construct downstream tasks, and obtain the accuracy of the multimodal large model to be evaluated under the original time-series multimodal data;

[0012] Define the timing consistency loss L temp The loss function L that causes the multimodal large model to output incorrect results is defined according to the attack type. adv According to the time consistency loss L temp and loss function L adv Generate the objective function;

[0013] Adversarial perturbations are added to frames of the original time-series multimodal data to optimize the adversarial perturbations with the goal of minimizing the objective function value, while keeping the difference between adversarial perturbations between adjacent time steps within a preset constraint.

[0014] Based on the optimized adversarial perturbation and the original temporal multimodal data, temporal adversarial samples are generated, and the imperceptibility of the perturbation is calculated.

[0015] The error output ratio (ASR) is obtained based on the output of the multimodal large model under multiple temporal adversarial examples.

[0016] Based on the internal states of the multimodal large model under multiple temporal adversarial examples, the temporal robustness index R is obtained. temp ;

[0017] Based on the error output ratio (ASR) and the timing robustness index (R) temp And the imperceptibility of disturbances, generating evaluation results.

[0018] Secondly, embodiments of this specification provide a temporal adversarial sample security evaluation system for multimodal large models, including:

[0019] The access module obtains the data modalities of the multimodal large model to be evaluated, constructs downstream tasks, and obtains the accuracy of the multimodal large model to be evaluated under the original time-series multimodal data.

[0020] Preparation module, defining timing consistency loss L temp The loss function L that causes the multimodal large model to output incorrect results is defined according to the attack type. adv According to the time consistency loss L tempand loss function L adv Generate the objective function;

[0021] The optimization module adds adversarial perturbations to the frames of the original time-series multimodal data to optimize the adversarial perturbations with the goal of minimizing the objective function value, while keeping the difference between adversarial perturbations between adjacent time steps within a preset constraint.

[0022] The generation module generates temporal adversarial samples based on the optimized adversarial perturbation and the original temporal multimodal data, and calculates the imperceptibility of the perturbation.

[0023] The evaluation module obtains the error output ratio (ASR) based on the output of the multimodal large model under multiple temporal adversarial examples.

[0024] The localization module obtains the temporal robustness index R based on the internal states of the multimodal large model under multiple temporal adversarial examples. temp ;

[0025] The results module, based on the error output ratio (ASR) and the timing robustness index (R), temp And the imperceptibility of disturbances, generating evaluation results.

[0026] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0027] In several embodiments of this specification, the provided method and system for temporal adversarial sample security evaluation for multimodal large models introduces temporal consistency loss, including two modes: inter-frame difference and optical flow guidance. This effectively constrains the dynamic coherence of adversarial perturbations in the temporal dimension, generating adversarial samples that more closely resemble real-world attack scenarios, thus improving the realistic threat and effectiveness of security evaluation. By analyzing the temporal similarity of the video coding representation within the model, the predictive stability of the model in the temporal dimension can be quantitatively evaluated, and its vulnerable time periods can be further located. Evaluation results are generated from three dimensions: attack effectiveness, temporal stability, and stealth, providing more valuable diagnostics for model security. By optimizing adversarial perturbations, while ensuring a high attack success rate, flickering or unnatural phenomena caused by frame-by-frame independent optimization are avoided, enabling the efficient generation of high-quality temporal adversarial samples.

[0028] Other features and advantages of various embodiments of this specification will be further revealed in the following detailed description and accompanying drawings. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram illustrating the security evaluation of temporal adversarial samples provided in this specification.

[0031] Figure 2 This is a schematic diagram of the security evaluation method for temporal adversarial samples provided in this specification.

[0032] Figure 3 This is a schematic diagram illustrating the timing robustness index provided in this specification.

[0033] Figure 4 This is a schematic diagram of the temporal adversarial sample security evaluation system provided in this specification. Detailed Implementation

[0034] The technical solutions of the embodiments of this specification will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of this specification and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of this specification.

[0035] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0036] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to facilitate the description of the embodiments and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this specification.

[0037] All data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0038] Before introducing the technical solutions described in this manual, the application scenarios and related technologies of the technical solutions will be introduced.

[0039] Multimodal Large Language Models (MLLMs) are advanced artificial intelligence systems capable of simultaneously understanding and generating multiple information modalities (such as text, images, video, and audio). These models are based on large-scale language models and achieve joint modeling of cross-modal semantics by integrating multimodal perception modules such as visual encoders and audio encoders. Typical applications include complex tasks such as video question answering, image-text generation, voice command control, and cross-modal retrieval. The core advantage of MLLMs lies in their powerful generalization and contextual reasoning capabilities, enabling them to align information from different modalities within a unified semantic space, thereby achieving human-level multimodal understanding and interaction.

[0040] However, precisely because of its high integration and black-box nature, MLLM faces severe security challenges in practical deployments. On the one hand, the model is extremely sensitive to input perturbations; even minute, imperceptible adversarial perturbations can cause its output to deviate significantly from the correct result. On the other hand, multimodal fusion mechanisms may introduce new vulnerabilities. For example, applying perturbations only to video frames could mislead speech understanding or text generation, forming cross-modal attack paths. These issues have received considerable attention in static image or pure text models, but they are more complex and have not been fully studied in dynamic multimodal scenarios involving the time dimension (such as video + speech).

[0041] Systematic testing of multimodal large language models is of great significance. First, testing can objectively assess the model's robustness against real-world interference or malicious attacks, avoiding misjudgments of reliability due to the "illusion of high accuracy." Second, by constructing multidimensional evaluation metrics covering attack success rate, perturbation concealment, and temporal stability, weaknesses in the model can be accurately identified in specific tasks, time periods, or modal interactions, providing targeted guidance for subsequent hardening. Third, in the context of increasingly stringent AI regulation, comprehensive security testing has become a crucial step in meeting compliance requirements and fulfilling algorithmic transparency and accountability obligations. More importantly, high-quality testing methods themselves can drive the development of defense technologies, guiding MLLM from pursuing performance metrics to considering security, reliability, and trustworthiness.

[0042] However, current testing of multimodal large language models mainly focuses on static single-modal scenarios, such as adding adversarial perturbations to independent images to attack the visual-language model, or replacing keywords in fixed text to interfere with language understanding. In real-world applications such as video and audio, information is presented as a continuous time series, with complex spatiotemporal relationships and dynamic interactions between modalities. This specification provides a temporal adversarial sample security evaluation method and system for multimodal large models. By introducing temporal consistency loss, including inter-frame differencing and optical flow guidance modes, it can effectively constrain the dynamic coherence of adversarial perturbations in the time dimension. The generated adversarial samples are closer to real-world attack scenarios, improving the realistic threat level and effectiveness of security evaluation. Please refer to the appendix. Figure 1 First, based on the original temporal multimodal data X, a temporal adversarial example generation module generates temporal adversarial examples Xadv. The temporal adversarial examples Xadv are then input into the large multimodal model to be evaluated, and feedback from the model is obtained. Based on the feedback from the large multimodal model, a multi-dimensional security evaluation is performed, resulting in a security evaluation report. The security evaluation report provided in this application can be used to guide subsequent model improvements or to conduct adversarial training.

[0043] Specifically, this specification first provides a method for evaluating the security of temporal adversarial examples for large multimodal models. Please refer to the appendix. Figure 2 The steps include:

[0044] Step S1) Obtain the data modalities of the multimodal large model to be evaluated, construct downstream tasks, and obtain the accuracy of the multimodal large model to be evaluated under the original time-series multimodal data.

[0045] The downstream task is selected from video question answering, time sequence event description generation, voice command understanding, cross-modal retrieval, action recognition, or multimodal dialogue. The corresponding input format, output format, and output correctness judgment rules are configured according to the selected downstream task.

[0046] Data modalities include images, audio, video, text, depth maps, and optical flow. Typical combinations in large multimodal models include "text + image," "audio + video," and "text + speech + vision." In temporal scenarios, video and audio naturally possess a temporal dimension and are often used in conjunction with text instructions or descriptions. Downstream tasks serve as benchmark scenarios for security evaluation.

[0047] The video question answering task requires the model to understand the visual dynamics, object interactions, or event development within a video clip and answer questions posed in natural language. Input format: A video clip and a natural language question related to the video content. Output format: A natural language answer, which can be a phrase, word, or complete sentence. Correctness judgment rule: The model-generated answer is compared with a human-annotated standard answer.

[0048] The temporal event description generation task aims to automatically generate a coherent, accurate, and logically sequential natural language description based on a sequence of events or actions that evolve over time in a video. Input format: An unannotated video clip (usually a continuous frame sequence, possibly containing audio). Output format: One or more sentences of natural language text describing the main events that occur in the video and their temporal order. Correctness assessment rules: Automatic text evaluation metrics are used to measure the semantic and keyword coverage similarity between the generated description and the reference description; human scoring can also be incorporated to assess fluency and factual consistency.

[0049] The voice command understanding task focuses on parsing the intent of a user's voice commands, potentially incorporating visual context (such as screen images or ambient video) to aid understanding. Input format: A voice signal (which can be transcribed into text or retain the original waveform / spectrum), and optional simultaneous visual information (such as a screenshot of the current interface or a scene video). Output format: Structured intent labels (such as "play music," "turn up brightness") or semantic parsing results with slot filling (such as {action: "set_timer", duration: "5 minutes"}). Correctness judgment rules: For classification tasks, accuracy is used to determine whether the predicted intent matches the label; for tasks with slot filling, both the intent category and key parameters must be completely matched.

[0050] Cross-modal retrieval aims to retrieve the most relevant content from a database of another modality (such as video or images) given a query in one modality (such as text), or vice versa. Input format: A query term (such as a text description) and a candidate database (such as multiple video clips); or conversely, video as the query and text as the candidates. Output format: A candidate list sorted by relevance, or directly returning the index of the single most matching result. Correctness judgment rules: Recall@K (such as R@1, R@5) or MedianRank are commonly used to measure retrieval performance, i.e., whether the correct match appears in the top K results; ranking relevance metrics such as nDCG can also be calculated.

[0051] Action recognition is a task that identifies specific categories of human actions or activities from video (sometimes combined with audio or sensor data). Input format: A video containing human behavior (usually a fixed-length frame sequence), possibly supplemented with audio or optical flow information. Output format: Predefined action category labels (e.g., "running," "clapping," "falling"). Correctness judgment rule: Classification accuracy (Top-1 or Top-5 Accuracy) is used to determine whether the predicted label matches the true label.

[0052] Multimodal dialogue tasks require dialogue systems to generate context-aware, informative, and natural responses when user input includes multiple modalities (e.g., image + text, voice + video). Input format: Multi-turn dialogue history (text) + current multimodal input (e.g., an image + user's voice or text question). Output format: Natural language response, which may include references, inferences, or suggestions regarding the image content. Correctness judgment rules: Difficult to fully automate; typically, a combination of automated metrics (e.g., BLEU, BERTScore) and human evaluation (scoring based on relevance, informativeness, fluency, factual consistency, etc.) is used; task-oriented judgment criteria (e.g., whether the user was successfully guided to complete the operation) can also be designed.

[0053] The accuracy of the multimodal large model to be evaluated is obtained under the original temporal multimodal data. This is used to compare the error output ratio (ASR) of the multimodal large model to be evaluated under temporal adversarial examples, thereby evaluating the accuracy of the multimodal large model to be evaluated under temporal adversarial examples.

[0054] Step S2) Define the timing consistency loss L temp The loss function L that causes the multimodal large model to output incorrect results is defined according to the attack type. adv According to the time consistency loss L temp and loss function L adv Generate the objective function.

[0055] Define the timing consistency loss L temp At that time, the difference in resistance to disturbances between adjacent time steps is calculated, and the difference is weighted and summed to obtain the time consistency loss L. temp Wherein, the adversarial perturbation is a sequence of noise vectors {δt} applied to each time step of the original temporal multimodal data, t∈[1,T], where T is the total number of time steps, and the temporal consistency loss L temp The formula for calculation is:

[0056]

[0057] Where p∈{1,2} and q∈{1,2}.

[0058] Temporal consistency loss L temp This is used to suppress abrupt jumps in adversarial perturbations between adjacent time steps, thereby avoiding the generation of obvious artificial traces such as flickering, jittering, or audio popping. Specifically, suppose the original temporal multimodal data (such as a video frame sequence or speech waveform) contains T time steps. At each time step t, an attacker applies a learnable noise vector δt, forming a perturbation sequence {δ1, δ2, ..., δT}. temp Overall temporal inconsistency is measured by calculating the difference norm between adjacent perturbations and then summing them with weights.

[0059] Here, p controls the norm type of the difference vector (p=2 emphasizes smooth changes in Euclidean space, or p=1 enhances robustness to anomalous jumps), while q determines whether to square the norm result (q=2) or preserve linearity (q=1). For example, in video adversarial attacks, if δt represents the pixel perturbation superimposed in frame t, then L temp It will penalize abrupt changes in inter-frame perturbations. For example, if a high-amplitude noise suddenly appears in frame 5 while frames 4 and 6 are almost perturbation-free, this discontinuity will significantly increase L. temp The value is thus suppressed during the optimization process. Similarly, in the speech instruction comprehension task, if δt acts on the time slice of the audio spectrogram, L temp It can prevent spectral disturbances from oscillating violently within adjacent time windows, maintaining the natural listening experience of speech.

[0060] On the other hand, in another, more refined implementation, the timing consistency loss L is defined. temp At that time, optical flow estimation is used to extract the optical flow field between adjacent frames of the original temporal multimodal data, characterizing the pixel-level motion information in the video. Motion compensation is performed on the adversarial perturbation δt at time step t, and it is remapped to the coordinate system of time step t+1. Based on the adversarial perturbation at time step t+1... The difference yields the time-series consistency loss L temp Wherein, the adversarial perturbation is a sequence of noise vectors {δt} applied to each time step of the original temporal multimodal data, t∈[1,T], where T is the total number of time steps, and the temporal consistency loss L temp The formula for calculation is:

[0061]

[0062] Wherein, the vector form of the vt optical flow field, Let p represent the perturbation obtained by remapping δt according to the optical flow vt, where p∈{1,2} and q∈{1,2}.

[0063] Temporal consistency loss L temp The construction no longer relies solely on the static difference of perturbations between adjacent time steps, but instead incorporates the dynamic structural information of the video content itself. Specifically, it captures pixel-level motion relationships between original video frames through optical flow estimation, and uses this to align and compensate for motion-aware adversarial perturbations. This strategy effectively solves the "apparent discontinuity" problem caused by object or camera motion, ensuring that perturbations remain coherent in a semantic space consistent with physical motion, thereby generating more covert and realistic temporal adversarial examples.

[0064] The original video contains T frames, denoted as {I1, I2, ..., I... T}, the attacker in each frame It The counter-perturbation δt is superimposed to form the perturbation sequence {δ1,δ2, ...,δt}. T If we directly calculate ||δ t+1 -δ t This approach ignores pixel position shifts caused by object movement, rotation, or camera translation between frames, leading to misinterpretations of inconsistencies even when the perturbation itself moves with the object. Therefore, an optical flow estimation algorithm (such as RAFT or FlowNet) is first used to estimate pixel positions from the original frame I. t and I t+1 Extracting optical flow vt from I, optical flow vt describes I t Each pixel moves to I during the time interval from t to t+1. t+1 The target location is then determined. Subsequently, the perturbation δt is remapped according to the optical flow vt to obtain... This involves projecting the disturbance at time t onto the coordinate system at time t+1, following the motion trajectory. Then, the disturbance δ actually applied to frame t+1 is calculated. t+1 The difference between them serves as a measure of local consistency at that time step.

[0065] Taking a video of a pedestrian walking as an example: Suppose that in frame t, an adversarial perturbation is added to the pedestrian's clothing area; due to the pedestrian moving forward, the same clothing area appears on the right side of the image in frame t+1. Without motion compensation, δ t and δ t+1 With almost no overlap at fixed pixel coordinates, L temp The value is artificially inflated; however, through the warp operation guided by optical flow, δ t If it is correctly pushed to the new position corresponding to frame t+1, then if δ t+1 If they are also concentrated in this area, then the difference between the two is very small, L temp Maintaining a low position indicates that the disturbance continues naturally as the target moves.

[0066] On the other hand, the loss function L that causes the multimodal large model to output incorrect results is defined according to the attack type. adv The methods include:

[0067] When employing a non-targeted attack, the true label y corresponding to the original temporal multimodal data is used. true Define the loss function L adv ,

[0068]

[0069] When employing a targeted attack, based on the preset erroneous target label y target Define the loss function L adv This is the negative logarithm of the multimodal large model's predicted probability of the target label, i.e.:

[0070]

[0071] Where Xadv is the temporal multimodal input with adversarial perturbation added, M is the large multimodal model to be evaluated, and P(▪|Xadv;M) represents the probability distribution of the output of the large multimodal model M to be evaluated for a specified class under the input Xadv.

[0072] Loss function L adv The design of the attack directly determines its direction and intensity, and its form needs to be defined according to the attack type. Attacks can be either non-targeted or targeted, each serving different security evaluation purposes. Non-targeted attacks aim to disrupt the model's normal judgment capabilities to the greatest extent possible, while targeted attacks attempt to induce the model to output specific erroneous results to test its stability in the face of carefully crafted misleading information.

[0073] In non-targeted attacks, the attacker doesn't care what specific incorrect answers the model outputs; they only want it to stop predicting the correct label. In this case, L adv Defined as the model's performance on the true label y under adversarial example Xadv. true The negative logarithm of the predicted probability. By minimizing this loss (equivalent to maximizing the negative log-likelihood), the optimization process continuously weakens the model's confidence in the correct answer. For example, in an action recognition task, if the original video shows a "jumping" action, the true label y... true For the "jump" option, the attacker adjusts the perturbation δ using methods such as gradient ascent, causing the model's output probability for "jump" to drop sharply. Even if the final prediction is "run", "walk", or any other arbitrary category, as long as it is not "jump", the attack is considered successful. This attack method is often used to evaluate the upper bound of the overall robustness of a model.

[0074] Targeted attacks are more specific and challenging. The attacker pre-specifies a flawed target label y. target (For example, forcibly guiding the "opening the door" action to "waving"), and designing L adv To minimize the negative log probability of the model with respect to the target label, we can maximize P(y). target |Xadv;M). In this way, the optimization process actively "pulls" the model output toward a preset error direction. For example, in a video question-and-answer scenario, the original question is "What is the person holding?", the correct answer is "water glass", but the attacker sets y target For "mobile phone". Through optimization of L adv The adversarial perturbation was carefully tuned so that the multimodal large model would answer "phone" with high confidence after watching the perturbed video. This type of attack not only verifies whether the model is easily misled, but also reveals its potential defects in semantic confusion, cross-modal alignment bias, and other aspects.

[0075] Step S3) Add adversarial perturbations to the frames of the original time-series multimodal data to optimize the adversarial perturbations with the objective function being minimized, while keeping the difference between adversarial perturbations between adjacent time steps within a preset constraint.

[0076] Regardless of the attack type used, Xadv is always composed of the original temporal multimodal input X (such as video frame sequences, audio waveforms, and their synchronized text) superimposed with a learnable perturbation δ, i.e., Xadv = X + δ, where M represents the large multimodal model to be evaluated. It is worth noting that, under the multimodal setting, y true or y target This could be a text answer, action category, retrieval index, or structured intent; therefore, P(·|Xadv;M) needs to be adapted to the specific output header of the model (such as a classification layer, language generation distribution, or embedding similarity). By using L... adv With timing consistency loss L temp By combining these methods, the effectiveness of the attack can be guaranteed while maintaining the naturalness of the perturbation in the time dimension, thereby generating adversarial samples that are both covert and efficient, providing a reliable tool for comprehensively evaluating the security boundaries of multimodal large models.

[0077] Step S4) Generate temporal adversarial samples based on the optimized adversarial perturbation and the original temporal multimodal data, and calculate the imperceptibility of the perturbation.

[0078] Methods for calculating the imperceptibility of disturbances include:

[0079] For raw time-series multimodal data Corresponding temporal adversarial examples A time-step comparison is performed, the perceived quality index for each time step is calculated, and the perceived quality index for all time steps is averaged to obtain the overall perturbation imperceptibility score.

[0080] The perceived quality index is either peak signal-to-noise ratio or structural similarity.

[0081] The goal of perturbation imperceptibility is to verify whether the generated adversarial perturbations are sufficiently covert at the level of human perception, that is, to ensure that the adversarial sample is highly similar to the original data visually or audibly, thereby guaranteeing the "imperceptibility" of the attack. "Imperceptibility" is a key indicator for measuring the practicality of adversarial attacks and the authenticity of security threats.

[0082] Specifically, the calculation of perturbation imperceptibility employs a time-step comparison approach. That is, for each time step t, the original frame (or audio clip, sensor reading, etc.) Xt is compared with its adversarial version Xt_adv, calculating an index reflecting perceptual quality. Finally, the arithmetic mean of the index values ​​over all T time steps is taken to obtain the overall imperceptibility score. Commonly used perceptual quality indices include Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), which characterize the degree of distortion from the perspectives of signal error and structural fidelity, respectively.

[0083] Taking video modality as an example, if Xt is the RGB image of frame t, and Xt_adv is the version after adding adversarial noise, then PSNR is evaluated by calculating the mean square error (MSE) and combining it with the maximum pixel value (usually 255). The calculation formula is PSNR = 10 × log 10 (MAX 2 / MSE). The higher the PSNR value (usually >30 dB), the weaker the disturbance, and the harder it is for the human eye to detect. SSIM, on the other hand, is closer to the human visual system, taking into account the local similarity of brightness, contrast, and structure. Its value ranges from [−1, 1], and the closer the value is to 1, the more consistent the two images are in structure. For example, in a traffic monitoring video, if the adversarial disturbance only introduces a small amount of noise at the edges of the vehicles, the PSNR may be 35 dB, and the SSIM may be 0.96, indicating that the disturbance is almost invisible while maintaining the semantic integrity of the image.

[0084] On the other hand, in the audio modality, PSNR can also be used for waveform domain evaluation, but perceptual metrics in the spectral domain are more commonly used. In this embodiment, either PSNR or SSIM (which can be applied to time-frequency representations such as Mel spectrograms) is used uniformly, which can still provide a consistent evaluation benchmark across modalities. It is worth noting that although PSNR is simple to calculate and widely used, it is not sensitive to structural distortion; while SSIM, although more consistent with visual perception, may fluctuate significantly in areas of intense motion or complex textures. Therefore, in actual evaluation, both are often used simultaneously, or one is selected as the primary metric based on the characteristics of the task.

[0085] Step S5) Obtain the error output ratio (ASR) based on the output of the multimodal large model to be evaluated under multiple temporal adversarial examples.

[0086] Error Output Ratio (ASR) is the proportion of incorrect outputs generated under multiple adversarial sample inputs. Multiple temporal adversarial samples are sequentially input into the multimodal large model M to be evaluated, and their corresponding outputs are recorded. For each adversarial sample Xadv, whether the model output is incorrect is determined according to the correctness judgment rules of its downstream task.

[0087] Step S6) Obtain the temporal robustness index R based on the internal states of the multimodal large model to be evaluated under multiple temporal adversarial examples.temp .

[0088] The video-coded representation Ft of a frame at one time step of the multimodal large model to be evaluated against the temporal adversarial example is used as the internal state, and the temporal robustness index R is obtained based on the video-coded representation Ft. temp ,

[0089]

[0090] Sim() represents the vector similarity between two vectors, such as using cosine similarity.

[0091] Time series robustness index R temp This is an evaluation dimension based on the stability of the model's internal representation. It is used to assess whether a large multimodal model can maintain reasonable temporal continuity in its internal video coding representation when faced with adversarial perturbations. Unlike focusing solely on the final output error rate, R... temp By delving into the model's internal workings, we can capture whether its semantic evolution of dynamic visual content suffers abnormal breaks or distortions due to minor perturbations, thereby revealing the vulnerability of multimodal large language models at the temporal modeling level.

[0092] Please see the appendix Figure 3 When the multimodal large model M to be evaluated processes a temporal adversarial example containing T frames, its video encoder (such as ViT, TimeSformer, or a 3D CNN backbone) outputs a high-dimensional feature vector at each time step t, namely the video encoded representation Ft. Ft is the model's internal semantic representation of that frame and its context. Even if the input is contaminated by adversarial noise, Ft still reflects what the model sees and understands. Ideally, if the video content itself has smooth motion or semantic evolution (such as a person slowly raising their arm), then the representations Ft of adjacent frames will be similar to Ft. t+1 They should be highly similar; however, if the adversarial perturbation disrupts the model's normal perception of temporal dynamics (such as causing a frame to be misjudged as a violent flicker or an irrelevant object), then abnormal discontinuities may occur between adjacent representations.

[0093] On the other hand, the temporal robustness index R is calculated based on the video coding representation Ft of a frame at one time step of the temporal adversarial example of the multimodal large model to be evaluated. temp The low-value region, i.e., the obtained value For time intervals below the preset reference value, the vulnerable links of the multimodal large model to be evaluated on the time axis are located based on the low value region.

[0094] Time series robustness index R tempThis internal temporal consistency is quantified by calculating the average similarity between the video coded representations of all adjacent time steps. Taking an adversarial video of "chopping vegetables" as an example: the movements in the original video are smooth and coherent, the model's internal representation Ft changes smoothly over time, and R... temp The similarity is close to 0.95. However, after applying adversarial perturbations, although the difference is difficult for the human eye to perceive (PSNR>32dB, SSIM>0.96), the model may misidentify the tool as a "flame" in frame 15 due to noise interference, causing F15 to deviate significantly from F14 and F16 in the semantic space, and the similarity to plummet to below 0.3. This local break significantly lowers the overall R-squared. temp A value (e.g., dropping to 0.78), even if the final output answer is correct (ASR=0%), exposes the instability of the model's internal temporal modeling mechanism. In tasks such as action recognition or multimodal dialogue, R... temp A decrease in these values ​​often indicates that the model has lost key temporal clues.

[0095] Step S7) Based on the error output ratio (ASR) and timing robustness index (R) temp And the imperceptibility of disturbances, generating evaluation results.

[0096] When generating evaluation results, the effectiveness of adversarial attacks is reflected in the error output ratio (ASR), and the time-series robustness index (R) is used to generate evaluation results. temp The system generates evaluation results that reflect the predictive stability of the multimodal large model under evaluation in the time dimension, and generates evaluation results that reflect the concealment and real threat of adversarial examples based on the imperceptibility of perturbations.

[0097] Error Rate of Output (ASR) directly measures the effectiveness of adversarial attacks. A high ASR value (e.g., >80%) indicates that even small perturbations can easily mislead the model, suggesting that its decision boundary is overly sensitive to subtle changes in temporal multimodal inputs. temp This reflects the model's predictive stability and internal consistency over time. Even with a low ASR, if R... temp A significant decrease (e.g., from 0.92 in normal video to 0.65 in adversarial examples) also indicates a break or confusion in the understanding of dynamic semantics within the multimodal large language model. The imperceptibility of the perturbation determines the stealth and real-world threat level of the adversarial example. If PSNR > 35 dB and SSIM > 0.97, it means the perturbation is almost imperceptible to the human visual or auditory system, and the attack can be carried out without alerting the user.

[0098] Based on the evaluation results, different security levels can be identified. For example, high risk is defined as high ASR + high imperceptibility + low R. tempThe model is easily fooled, cannot maintain its internal temporal logic, and is difficult to defend against attacks. Medium risk is characterized by high ASR but visible perturbations, or low ASR but visible R... temp Low: The threat is limited by detectability and implies potential reasoning flaws. Low risk is defined as low ASR + high R. temp +High imperceptibility: The stability of the multimodal large language model indicates that the model has strong robustness and can resist covert temporal attacks.

[0099] On the other hand, this specification provides a temporal adversarial example security evaluation system for multimodal large models. Please refer to the appendix. Figure 4 ,include:

[0100] The access module 100 acquires the data modalities of the multimodal large model to be evaluated, constructs downstream tasks, and obtains the accuracy of the multimodal large model to be evaluated under the original time-series multimodal data.

[0101] Preparation module 200, defining the timing consistency loss L temp The loss function L that causes the multimodal large model to output incorrect results is defined according to the attack type. adv According to the time consistency loss L temp and loss function L adv Generate the objective function;

[0102] The optimization module 300 adds adversarial perturbations to the frames of the original time-series multimodal data to optimize the adversarial perturbations with the goal of minimizing the objective function value, while keeping the difference between adversarial perturbations between adjacent time steps within a preset constraint.

[0103] The generation module 400 generates temporal adversarial samples based on the optimized adversarial perturbation and the original temporal multimodal data, and calculates the imperceptibility of the perturbation.

[0104] The evaluation module 500 obtains the error output ratio (ASR) based on the output of the multimodal large model under multiple temporal adversarial examples.

[0105] The localization module 600 obtains the temporal robustness index R based on the internal states of the multimodal large model under multiple temporal adversarial examples. temp ;

[0106] Result module 700, based on the error output ratio (ASR) and timing robustness index (R). temp And the imperceptibility of disturbances, generating evaluation results.

[0107] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims of this specification.

Claims

1. A method for evaluating the security of temporal adversarial examples for multimodal large models, characterized in that, Including the following steps: Obtain the data modalities of the multimodal large model to be evaluated, construct downstream tasks, and obtain the accuracy of the multimodal large model to be evaluated under the original time-series multimodal data; Define the timing consistency loss L temp The loss function L that causes the multimodal large model to output incorrect results is defined according to the attack type. adv According to the time consistency loss L temp and loss function L adv Generate the objective function; Adversarial perturbations are added to frames of the original time-series multimodal data to optimize the adversarial perturbations with the goal of minimizing the objective function value, while keeping the difference between adversarial perturbations between adjacent time steps within a preset constraint. Based on the optimized adversarial perturbation and the original temporal multimodal data, temporal adversarial samples are generated, and the imperceptibility of the perturbation is calculated. The error output ratio (ASR) is obtained based on the output of the multimodal large model under multiple temporal adversarial examples. Based on the internal states of the multimodal large model under multiple temporal adversarial examples, the temporal robustness index R is obtained. temp ; Based on the error output ratio (ASR) and the timing robustness index (R) temp And the imperceptibility of disturbances, generating evaluation results.

2. The method for evaluating the security of temporal adversarial examples for multimodal large models according to claim 1, characterized in that, The downstream task is selected from video question answering, time sequence event description generation, voice command understanding, cross-modal retrieval, action recognition, or multimodal dialogue. The corresponding input format, output format, and output correctness judgment rules are configured according to the selected downstream task.

3. The method for evaluating the security of temporal adversarial examples for multimodal large models according to claim 1, characterized in that, Define the timing consistency loss L temp At that time, the difference in resistance to disturbances between adjacent time steps is calculated, and the difference is weighted and summed to obtain the time consistency loss L. temp Wherein, the adversarial perturbation is a sequence of noise vectors {δt} applied to each time step of the original temporal multimodal data, t∈[1,T], where T is the total number of time steps, and the temporal consistency loss L temp The formula for calculation is: ， Where p∈{1,2} and q∈{1,2}.

4. The method for evaluating the security of temporal adversarial examples for multimodal large models according to claim 1, characterized in that, Define the timing consistency loss L temp At that time, optical flow estimation is used to extract the optical flow field between adjacent frames of the original temporal multimodal data, characterizing the pixel-level motion information in the video. Motion compensation is performed on the adversarial perturbation δt at time step t, and it is remapped to the coordinate system of time step t+1. Based on the adversarial perturbation at time step t+1... The difference yields the time-series consistency loss L temp Wherein, the adversarial perturbation is a sequence of noise vectors {δt} applied to each time step of the original temporal multimodal data, t∈[1,T], where T is the total number of time steps, and the temporal consistency loss L temp The formula for calculation is: ， Wherein, the vector form of the vt optical flow field, Let p represent the perturbation obtained by remapping δt according to the optical flow vt, where p∈{1,2} and q∈{1,2}.

5. The method for evaluating the security of temporal adversarial examples for multimodal large models according to claim 1, characterized in that, The loss function L that causes the multimodal large model to output incorrect results is defined according to the attack type. adv The methods include: When employing a non-targeted attack, the true label y corresponding to the original temporal multimodal data is used. true Define the loss function L adv , ， When employing a targeted attack, based on the preset erroneous target label y target Define the loss function L adv This is the negative logarithm of the multimodal large model's predicted probability of the target label, i.e.: ， Where Xadv is the temporal multimodal input with adversarial perturbation added, M is the large multimodal model to be evaluated, and P(∙|Xadv;M) represents the probability distribution of the output of the large multimodal model M to be evaluated for a specified class under the input Xadv.

6. The method for evaluating the security of temporal adversarial examples for multimodal large models according to claim 1, characterized in that, Methods for calculating the imperceptibility of disturbances include: For raw time-series multimodal data Corresponding temporal adversarial examples A time-step comparison is performed, the perceived quality index for each time step is calculated, and the perceived quality index for all time steps is averaged to obtain the overall perturbation imperceptibility score. The perceived quality index is either peak signal-to-noise ratio or structural similarity.

7. The method for evaluating the security of temporal adversarial examples for multimodal large models according to claim 1, characterized in that, The video-coded representation Ft of a frame at one time step of the multimodal large model to be evaluated against the temporal adversarial example is used as the internal state, and the temporal robustness index R is obtained based on the video-coded representation Ft. temp , ， Sim() represents the vector similarity between two vectors.

8. The method for evaluating the security of temporal adversarial examples for multimodal large models according to claim 1, characterized in that, The temporal robustness index R is calculated based on the video coding representation Ft of a frame at one time step of the temporal adversarial example of the multimodal large model to be evaluated. temp The low-value region, i.e., the obtained value For time intervals below a preset reference value, the vulnerable links of the multimodal large model to be evaluated on the time axis are located based on the low value region.

9. The method for evaluating the security of temporal adversarial examples for multimodal large models according to claim 1, characterized in that, When generating evaluation results, the effectiveness of adversarial attacks is reflected in the error output ratio (ASR), and the time-series robustness index (R) is used to generate evaluation results. temp The system generates evaluation results that reflect the predictive stability of the multimodal large model under evaluation in the time dimension, and generates evaluation results that reflect the concealment and real threat of adversarial examples based on the imperceptibility of perturbations.

10. A temporal adversarial example security evaluation system for multimodal large models, characterized in that, include: The access module obtains the data modalities of the multimodal large model to be evaluated, constructs downstream tasks, and obtains the accuracy of the multimodal large model to be evaluated under the original time-series multimodal data. Preparation module, defining timing consistency loss L temp The loss function L that causes the multimodal large model to output incorrect results is defined according to the attack type. adv According to the time consistency loss L temp and loss function L adv Generate the objective function; The optimization module adds adversarial perturbations to the frames of the original time-series multimodal data to optimize the adversarial perturbations with the goal of minimizing the objective function value, while keeping the difference between adversarial perturbations between adjacent time steps within a preset constraint. The generation module generates temporal adversarial samples based on the optimized adversarial perturbation and the original temporal multimodal data, and calculates the imperceptibility of the perturbation. The evaluation module obtains the error output ratio (ASR) based on the output of the multimodal large model under multiple temporal adversarial examples. The localization module obtains the temporal robustness index R based on the internal states of the multimodal large model under multiple temporal adversarial examples. temp ; The results module, based on the error output ratio (ASR) and the timing robustness index (R), temp And the imperceptibility of disturbances, generating evaluation results.

Images