A multi-modal large model availability evaluation method based on repeated output

Abstract

The present application relates to the technical field of artificial intelligence, and provides a multi-modal large model availability evaluation method based on repeated output, an electronic device and a storage medium. In view of the problems of unstable extreme output capability (i.e. unable to stably trigger the maximum output length) caused by insufficient continuous control of the generation process and low single-sample efficiency and limited scale test caused by complex loss function in the prior art, the present application adds a slight disturbance to the original image (optimizes the target character sequence output probability through a projected gradient descent algorithm and meets the amplitude constraint), forms a disturbed image; the original text and the disturbed image are combined into a test sample to input a multi-modal large model, induce the self-recurrence of the multi-modal large model to generate a continuous output target sequence until a preset maximum length, realize a repeated output cycle; record the output length, hardware energy consumption and inference duration, and compare and evaluate the availability with the original sample without disturbance. The method effectively improves stability and efficiency and supports scale test.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and provides a method for evaluating the availability of multimodal large models based on repeated output. Background Technology

[0002] In recent years, multimodal large language models (hereinafter referred to as "multimodal large models") have made significant progress in cross-modal understanding and generation tasks by integrating visual and linguistic capabilities, and have been widely applied to tasks such as image description, video question answering, and visual reasoning. These models typically integrate a visual modality processor on top of a large-scale language model to achieve visual context-aware interaction. Therefore, multimodal large models usually contain billions of parameters, and the inference phase demands extremely high computational and energy consumption. Inference efficiency and resource utilization have become key factors affecting the actual availability of the system. In practical deployments, especially in scenarios with high real-time requirements (such as autonomous driving) or resource-constrained terminals (such as mobile devices), it is necessary to comprehensively evaluate the service availability of the model under extreme or abnormal inference loads in order to identify potential performance bottlenecks in advance and formulate effective mitigation strategies, thereby ensuring the stable and reliable operation of the system.

[0003] Multimodal large-scale models: Multimodal large-scale models refer to artificial intelligence models that can simultaneously process linguistic modalities (such as text) and visual modalities (such as images and videos), fuse information from different modalities for understanding, and ultimately generate corresponding text information. These models typically contain billions or even tens of billions of parameters. While possessing powerful semantic understanding and cross-modal reasoning capabilities, they place high demands on computational resources such as GPU memory, computation time, and energy consumption. Current multimodal large-scale model generation methods are mainly based on autoregressive mechanisms. That is, when generating text, the model depends on the content of the previously generated sequence character by character. The output result of each step affects the prediction of subsequent characters until a special terminator EOS is encountered or the preset maximum generation length is reached. This character-by-character generation mechanism makes the model's inference overhead closely related to the length of the generated sequence. The longer the output, the higher the computation time and energy consumption. Therefore, the output length directly affects the model's usability in different application scenarios.

[0004] Repeated Output Phenomenon: Research indicates that large multimodal models may fall into a "repeated output" phenomenon during natural generation, continuously generating the same or highly similar content, thus affecting the quality of the generated text. This phenomenon is closely related to the characteristics of autoregressive mechanisms. Since the model's predictions rely on historical context, once repeated content is generated, the repeated parts are incorporated into the context, further increasing the probability of that content reappearing in the future. This self-reinforcing effect continuously amplifies the repetitive trend, causing the model to fall into a cyclical repetitive generation process. This phenomenon not only reveals the inherent characteristics of autoregressive mechanisms but also provides a valuable feature for constructing usability assessment methods that simulate extreme resource consumption.

[0005] Currently, usability evaluation methods for multimodal large models mainly focus on extending the model's output process by delaying the emergence of EOS (Employment Optimization Images) to test its stability and robustness under high-energy-consumption and high-latency scenarios. For example, a recent study, Gao et al., proposed the Verbose Images method, which induces the model to generate longer text by constructing specific inputs to increase resource consumption. This method designs three different loss functions:

[0006] (1) Delayed EOS loss: Reduce the generation probability of the terminator EOS and delay the time when the model stops outputting;

[0007] (2) Character uncertainty loss: Fit the output distribution to a uniform distribution to increase the uncertainty of the generated sequence, thereby extending the sequence;

[0008] (3) Character diversity loss: Increase the rank of the hidden layer state matrix to increase the diversity and complexity of the output content, and further prolong the generation process. In addition, this method also introduces a time weight adjustment mechanism, which dynamically allocates the weights of the loss function to achieve a balance between delayed EOS, uncertainty and diversity objectives, so as to generate more suitable test samples.

[0009] Current methods for evaluating or testing models under high-resource-consumption conditions typically rely on delaying the appearance of a special end-of-sequence (EOS) character to trigger longer outputs, thereby increasing energy consumption and inference costs. These methods have the following limitations in practical applications:

[0010] Limited ability to trigger extreme outputs: The reliance on delayed terminators fails to structurally maintain continuous control over the generation process. As the generated sequence grows, its impact on the far-end output weakens, making it impossible to stably trigger the maximum output length of the model, thus making it difficult to cover more extreme resource consumption scenarios.

[0011] Evaluation efficiency and scalability are limited: Existing methods often require the design of complex loss functions or the optimization of samples with high overhead. The generation of a single sample is time-consuming and computationally expensive, making it difficult to run efficiently in batch or large-scale scenarios and hindering the implementation of extensive availability testing in real service environments.

[0012] To address the aforementioned issues, this invention proposes a multimodal large model availability assessment technique based on repeated output. This method leverages the autoregressive nature of the model generation process, designing a concise scheme to trigger and maintain repeated generation behavior, thereby constructing a controllable and efficient stress test sample generation process to evaluate the model's stability and service availability under high energy consumption and high latency conditions. Summary of the Invention

[0013] The purpose of this invention is to solve the problem of unstable extreme output capability (i.e., inability to stably trigger the maximum output length of the model) caused by insufficient continuous control over the generation process in existing multimodal large model availability evaluation techniques based on delayed terminators, as well as the problems of low single-sample generation efficiency and limited scalability testing caused by complex loss function design.

[0014] To achieve the above objectives, the present invention employs the following technical means:

[0015] This invention provides a method for evaluating the usability of a multimodal large model based on repeated outputs, comprising the following steps:

[0016] Step (a) Generate test samples: Obtain the original text input and the original image input; add a small perturbation to the original image to form a perturbation image, wherein the perturbation is optimized by the projection gradient descent algorithm to maximize the output probability of the target character sequence and satisfy the perturbation amplitude constraint; combine the original text input and the perturbation image to form test samples;

[0017] Step (b) Triggering a repetitive output loop: Input the test sample into the multimodal large model for inference generation, and induce the multimodal large model to continuously output the target character sequence during the autoregressive generation process until the preset maximum output length is reached, thus forming a repetitive output loop;

[0018] Step (c) Evaluate usability metrics: Record the output length, hardware power consumption and inference time of the multimodal large model in step (b), and compare them with the original sample without perturbation to generate usability evaluation results.

[0019] In the above scheme, the target character sequence consists of c characters randomly selected from the model vocabulary, where c is a preset positive integer. The repeated output loop is triggered by increasing the generation probability of the target character.

[0020] In the above scheme, the loss function in step (a) is defined as:

[0021]

[0022] in It is the output length. Representing the Generate the first output position at the nth output position The probability of a target character.

[0023] In the above scheme, the iterative update formula of the projection gradient descent algorithm is:

[0024]

[0025] in Represents the loss function pair The gradient of the step-time perturbation;

[0026] This indicates a sign function; when the internal value is positive, the result is 1; when the internal value is negative, the result is -1; and when the internal value is 0, the result is 0.

[0027] Indicates the update step size. The projection function restricts the disturbance to the original disturbance. The origin has a radius of In a circular space.

[0028] In the above scheme, the mechanism for triggering the repeated output loop in step (b) includes:

[0029] The target character sequence is incorporated into the historical context after it is first generated by the model;

[0030] The model continuously predicts the sequence based on its autoregressive properties, forming a self-reinforcing cycle with increased probability.

[0031] In the above scheme, the availability assessment results include:

[0032] Output length: The average length of text generated by the statistical model, reflecting the output trend of the model under extreme conditions;

[0033] Hardware power consumption: Detects the average hardware power consumption during a single inference process to assess the computational load;

[0034] Inference time: Measure the average response time of the model from input to output to examine its adaptability in latency-sensitive tasks.

[0035] The present invention also provides an electronic device, including a processor and a memory, the memory storing a computer program, wherein the processor executes the computer program to implement any of the methods described above.

[0036] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described herein.

[0037] Because the present invention employs the above-mentioned technical means, it has the following beneficial effects:

[0038] 1. The present invention solves the problem that the existing technology cannot stably trigger the maximum output length of the model due to insufficient continuous control over the generation process by triggering the repeated output loop in step (b). It achieves the effect of forming a probability-enhancing self-reinforcing loop in the process of inducing autoregressive generation of multimodal large models, and ensuring that the model can stably and fully reach the preset maximum output length.

[0039] Specifically, existing technologies (such as the Verbose Images method) rely on delaying the appearance of special end-of-sequence (EOS) markers to extend the output sequence, but this mechanism is only effective in the initial stage. As the generated sequence grows, its impact on the far-end output gradually diminishes, making it impossible to maintain continuous control and causing the model to struggle to stably trigger the maximum output length. This invention incorporates the target character sequence into the historical context through step (b) and utilizes the autoregressive properties of a multimodal large model to enable the model to continuously predict the target sequence after its initial generation, forming a probability-enhancing self-reinforcing loop—that is, the generated target character is repeatedly introduced into the context, further increasing its subsequent generation probability. This mechanism structurally achieves dynamic maintenance of the generation process: the initial trigger point ensures high-probability output of the target character through the perturbation image optimized in step (a), while the autoregressive properties extend local triggering into a global continuous loop, avoiding the fundamental defect in existing methods where control weakens as the sequence grows. Therefore, before reaching the preset maximum length, the model remains in a stable, high-intensity output state, effectively covering extreme resource-consuming scenarios.

[0040] 2. The present invention solves the problems of low efficiency in single sample generation and limited large-scale testing caused by complex loss functions in the prior art by simplifying the loss function design and optimizing the projection gradient descent algorithm in step (a). It achieves the effect of significantly reducing the computational overhead of test sample generation, improving the processing speed of single samples and supporting batch large-scale usability testing.

[0041] Specifically, existing methods (such as Verbose Images) require the design of multiple complex loss functions (including delayed EOS loss, character uncertainty loss, and character diversity loss) and optimization through costly dynamic weight adjustment mechanisms. This results in a computationally intensive and time-consuming single-sample generation process, making it unsuitable for batch processing. This invention introduces a single cyclic loss function in step (a). This design focuses solely on maximizing the output probability of the target character sequence, avoiding the complexity of multi-objective optimization. Simultaneously, it incorporates projective gradient descent for efficient perturbation iteration, with its sign function update and projection constraint mechanism ensuring lightweight computation and rapid convergence at each step. This simplifies the test sample generation process from relying on high-dimensional parameter tuning to single-variable optimization: the loss function focuses only on the cumulative increase in the target character probability (rather than multi-dimensional balancing), and projective gradient descent achieves rapid convergence of the perturbation while satisfying amplitude constraints, significantly reducing the number of iterations and computational cost. Therefore, the single-sample generation time is significantly shortened, and large-scale batch generation of test samples can be supported without additional resource allocation, completely solving the core bottleneck of limited large-scale testing in existing technologies. Attached Figure Description

[0042] Figure 1 This is a simplified flowchart of the usability assessment process for the present invention;

[0043] Figure 2 This is a simplified block diagram of the functional modules of the present invention; Detailed Implementation

[0044] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.

[0045] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.

[0046] To address the problem proposed in this invention, a multimodal large model usability assessment technique based on repeated outputs is proposed. This invention fully utilizes the self-reinforcing characteristic of repeated outputs under the autoregressive mechanism of multimodal large models to construct an efficient usability assessment method, the main features of which are as follows:

[0047] Simulating extreme resource consumption by using repeated output: By designing a loop-induced mechanism, the model is continuously trapped in repeated output during the generation process until the maximum output limit is reached, thereby significantly increasing inference energy consumption and response time, better simulating extreme resource consumption in real-world environments, and improving the comprehensiveness and realism of usability assessment.

[0048] A simplified cyclic loss function is used to improve efficiency: Unlike existing methods that rely on complex loss functions, this invention only needs to introduce a single cyclic loss function to induce the model to enter a repetitive state. The generation process has low computational overhead and short single-sample generation time, enabling it to run efficiently in batch and large-scale scenarios.

[0049] I. Overview:

[0050] This invention proposes a multimodal large model availability evaluation technique based on repeated output. This technique evaluates the performance of multimodal large models under extreme resource consumption conditions, involving indicators such as output length, hardware power consumption, and response time, thereby comprehensively analyzing the model's availability level under high load scenarios. This method can identify potential stability risks in advance, thereby improving the system's early warning capabilities and ensuring the continuous and stable operation of services. The core of this invention lies in how to construct test samples to effectively trigger the repeated output behavior of the multimodal large model and simulate extreme inference overhead. To this end, this invention constructs test samples by adding a small perturbation to the original image input and combining it with the original text input; then, it optimizes the perturbation using a specially designed loss function to induce the model to generate repeated content until the maximum output length is reached, thereby achieving effective evaluation of power consumption and response time.

[0051] II. Introduction to System Functional Modules:

[0052] Figure 2 The simplified block diagram of the functional modules of this invention illustrates the multimodal large model usability assessment technology based on repeated output, which mainly includes two stages: a test sample generation stage and a target model evaluation stage, forming a complete closed loop from test sample preparation to model usability evaluation. In the test sample generation stage, a test sample set capable of triggering repeated output is constructed by optimizing the perturbation through a loss function. In the target model evaluation stage, the test sample set is input into the target model, and the performance of various model metrics is detected and recorded to obtain the model's usability assessment results.

[0053] III. System Design Details:

[0054] Based on the above generation process, this section will provide a detailed introduction to the multimodal large model availability assessment technology based on repeated output, which is divided into the test sample generation stage and the target model evaluation stage.

[0055] 1. Test Sample Generation Stage

[0056] The input samples consist of text and image inputs. This invention applies a small perturbation to the image while preserving its original semantics to generate image input for evaluation. Since the energy consumption and inference time of a large multimodal model are primarily related to the length of the output sequence, the goal of this invention is to maximize the output length. Unlike previous methods that rely on delayed EOS characters, this method induces the model to enter a repetitive output loop, triggering continuous generation behavior, thereby better simulating resource consumption under extreme conditions. To this end, this invention proposes a dedicated loss function to optimize the perturbation. The optimization objective can be formalized as:

[0057]

[0058] in It is text input. It is the original image. This represents the perturbation added to the original image. It is the loss function. To constrain the intensity of the disturbance, by limiting the disturbance's... Norms ensure that the perturbation amplitude is so small that it is imperceptible to the human eye, thus better simulating malicious intent in real-world scenarios.

[0059] The core function of the loss function is to guide the model to gradually enter a state of repeated generation. The specific method is: first, randomly select from the model's vocabulary... A set of characters are selected as target characters; then, a loss function is used to encourage the model to generate these target characters. Since directly forcing the model to generate certain characters is difficult, this invention employs a targetless optimization strategy, increasing the output probability of the target characters to induce repetition. The loss function is defined as follows:

[0060]

[0061] in It is the output length. Representing the Generate the first output position at the nth output position The probability of a target character is calculated. By optimizing this loss function, the model can be made to tend to generate the target character at multiple output positions. Once these target characters are generated multiple times and form an initial loop, the model's autoregressive mechanism creates a self-reinforcing loop, where the generated target characters further increase their probability of being generated again, eventually leading to an infinite repetitive cycle.

[0062] Finally, to minimize the loss function and obtain the optimal perturbation, this invention employs the projected gradient descent algorithm for iterative optimization of the perturbation. During the update process, the... The optimization process for the step is as follows:

[0063]

[0064] in Represents the loss function pair The gradient of the step-time perturbation, The function indicates the sign of the function (the result is 1 when the internal value is positive; -1 when the internal value is negative; and 0 when the internal value is 0). Indicates the update step size. The projection function restricts the disturbance to the original disturbance. The origin has a radius of In a circular space.

[0065] Through several iterative steps, the perturbation is gradually optimized to reduce the value of the loss function and continuously increase the output probability of the target character. Once the model output falls into a state of repeated generation, optimization stops, and the original text input and the image input containing the perturbation are combined to form the final test sample.

[0066] 2. Target Model Evaluation Phase

[0067] After generating the test samples, a batch of test sample sets can be obtained. This sample set is then input into the target multimodal large model to measure and record various model metrics, specifically including:

[0068] Output length: The average length of text generated by the statistical model, reflecting the output trend of the model under extreme conditions;

[0069] Hardware power consumption: Detects the average hardware power consumption during a single inference process to assess the computational load;

[0070] Inference time: Measure the average response time of the model from input to output to examine its adaptability in latency-sensitive tasks.

[0071] During the evaluation process, the above indicators are compared with the results of the original, unperturbed test samples to obtain the performance degradation degree and availability of the target model under high resource consumption conditions. This method can provide a reference for the optimization and fault-tolerant design of the model in actual deployment.

[0072] Example 1

[0073] See Figure 1 This invention provides a method for usability evaluation of multimodal large models based on repeated outputs, comprising the following steps:

[0074] Step (a) Generate test samples: Obtain the original text input and the original image input; add a small perturbation to the original image to form a perturbation image, wherein the perturbation is optimized by the projection gradient descent algorithm to maximize the output probability of the target character sequence and satisfy the perturbation amplitude constraint; combine the original text input and the perturbation image to form test samples;

[0075] Step (b) Triggering a repetitive output loop: Input the test sample into the multimodal large model for inference generation, and induce the multimodal large model to continuously output the target character sequence during the autoregressive generation process until the preset maximum output length is reached, thus forming a repetitive output loop;

[0076] Step (c) Evaluate usability metrics: Record the output length, hardware power consumption and inference time of the multimodal large model in step (b), and compare them with the original sample without perturbation to generate usability evaluation results.

[0077] In the above scheme, the target character sequence consists of c characters randomly selected from the model vocabulary, where c is a preset positive integer. The repeated output loop is triggered by increasing the generation probability of the target character.

[0078] In the above scheme, the loss function in step (a) is defined as:

[0079]

[0080] in It is the output length. Representing the Generate the first output position at the nth output position The probability of a target character.

[0081] In the above scheme, the iterative update formula of the projection gradient descent algorithm is:

[0082]

[0083] in Represents the loss function pair The gradient of the step-time perturbation;

[0084] This indicates a sign function; when the internal value is positive, the result is 1; when the internal value is negative, the result is -1; and when the internal value is 0, the result is 0.

[0085] Indicates the update step size. The projection function restricts the disturbance to the original disturbance. The origin has a radius of In a circular space.

[0086] In the above scheme, the mechanism for triggering the repeated output loop in step (b) includes:

[0087] The target character sequence is incorporated into the historical context after it is first generated by the model;

[0088] The model continuously predicts the sequence based on its autoregressive properties, forming a self-reinforcing cycle with increased probability.

[0089] In the above scheme, the availability assessment results include:

[0090] Output length: The average length of text generated by the statistical model, reflecting the output trend of the model under extreme conditions;

[0091] Hardware power consumption: Detects the average hardware power consumption during a single inference process to assess the computational load;

[0092] Inference time: Measure the average response time of the model from input to output to examine its adaptability in latency-sensitive tasks.

[0093] Example 2

[0094] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in Embodiment 1.

[0095] Example 3

[0096] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0097] In summary, the present invention has the following characteristics:

[0098] 1. A Usability Assessment Method Based on Repeated Output Mechanism: This invention proposes for the first time to utilize the self-reinforcing repetition phenomenon in the autoregressive generation process of multimodal large models. By optimizing the perturbation-induced model to enter a repeated output loop, it maintains output for a long time before reaching the maximum output length, thereby effectively simulating high-energy-consumption and high-latency scenarios under extreme conditions, and thus achieving the assessment of the usability (such as resource consumption, stability, and response performance) of multimodal large models. Compared with existing methods that rely on delayed EOS characters, this method can trigger extreme output states more stably and fully.

[0099] 2. Low-Overhead, High-Efficiency Test Sample Generation Mechanism: This invention proposes a simplified loss function design, combined with the projective gradient descent algorithm, to efficiently generate batches of test samples using only a single loss function while maintaining imperceptible perturbations. Unlike existing optimization methods with complex loss functions and high computational overhead, this invention reduces the time and computational cost of test sample generation, making it more practical in large-scale, automated testing scenarios.

Claims

1. A method for evaluating the usability of a multimodal large model based on repeated outputs, characterized in that, Includes the following steps: Step (a) Generate test samples: Obtain the original text input and the original image input; add a small perturbation to the original image to form a perturbation image, wherein the perturbation is optimized by the projection gradient descent algorithm to maximize the output probability of the target character sequence and satisfy the perturbation amplitude constraint; combine the original text input and the perturbation image to form test samples; Step (b) Triggering a repetitive output loop: Input the test sample into the multimodal large model for inference generation, and induce the multimodal large model to continuously output the target character sequence during the autoregressive generation process until the preset maximum output length is reached, thus forming a repetitive output loop; Step (c) Evaluate usability metrics: Record the output length, hardware power consumption and inference time of the multimodal large model in step (b), and compare them with the original sample without perturbation to generate usability evaluation results; The iterative update formula for the projection gradient descent algorithm is as follows: in Represents the loss function pair The gradient of the step-time perturbation; This indicates a sign function; when the internal value is positive, the result is 1; when the internal value is negative, the result is -1; and when the internal value is 0, the result is 0. Indicates the update step size. The projection function restricts the disturbance to the original disturbance. The origin has a radius of In a circular space; The mechanism that triggers the repetitive output loop in step (b) includes: The target character sequence is incorporated into the historical context after it is first generated by the model; The model continuously predicts the sequence based on its autoregressive properties, forming a self-reinforcing cycle with increased probability.

2. The method according to claim 1, characterized in that, The target character sequence consists of c characters randomly selected from the model vocabulary, where c is a preset positive integer. The repeated output loop is triggered by increasing the generation probability of the target character.

3. The method according to claim 1, characterized in that, The loss function in step (a) is defined as: in It is the output length. Representing the Generate the first output position at the nth output position The probability of a target character.

4. The method according to claim 1, characterized in that, The availability assessment results include: Output length: The average length of text generated by the statistical model, reflecting the output trend of the model under extreme conditions; Hardware power consumption: Detects the average hardware power consumption during a single inference process to assess the computational load; Inference time: Measure the average response time of the model from input to output to examine its adaptability in latency-sensitive tasks.

5. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method of any one of claims 1-4.

Images