A multi-modal large language model inference method based on parallel visual token scheduling

Abstract

The application discloses a multimodal large language model inference method based on parallel visual token scheduling, comprising the following steps: S1, obtaining input information of the model, the input information comprising image data and text query data input by a user; dividing visual tokens by using attention weights of category labels in a visual encoder, and dividing visual input into subject tokens and non-subject tokens; S2, constructing two parallel input streams, inputting the two types of visual tokens and the same text tokens into the first n layers of a Transformer layer of a multimodal large language model for processing; S3, in a shallow stage, the multimodal large language model completes migration of visual information to text tokens by using a self-attention mechanism; S4, after the n-th layer, fusing text token representations in the two branches, discarding the non-subject tokens, and only retaining the subject tokens to enter the remaining Transformer layers for inference until the multimodal large language model finally generates and outputs a target output result.

Description

Technical Field

[0001] This invention belongs to the field of computer model processing technology, specifically relating to a multimodal large language model inference method based on parallel visual token scheduling. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large language models (MLLMs) that integrate image and language capabilities have demonstrated powerful understanding and reasoning abilities in tasks such as image-text question answering, image description generation, and assisted perception. A typical multimodal large language model consists of three parts: a visual encoder, a cross-modal projector, and a pre-trained language model. It can transform high-resolution image input into tens of thousands of visual tags, which then participate in reasoning along with text tags.

[0003] While the reasoning capabilities of multimodal models continue to improve, their computational costs have also increased significantly, primarily due to the quadratic growth of the self-attention mechanism in the Transformer architecture as the length of the input sequence increases. In high-resolution image input scenarios, visual markers often occupy a large portion of the input sequence, severely slowing down model inference speed. This is particularly pronounced in applications such as mobile augmented reality and interactive question-answering systems, where response time is critical.

[0004] To address these issues, researchers have proposed a series of visual label compression and pruning techniques to reduce inference costs. Existing methods attempt to select highly important labels through attention scoring or merge redundant labels based on semantic similarity. However, these methods either rely on manually set heuristic rules, resulting in insufficient generalization, or introduce additional modules and training processes, increasing system complexity and engineering costs.

[0005] Therefore, there is an urgent need for a technical solution that can make full use of visual information in the early stages of inference and achieve efficient labeling and pruning without increasing training costs, so as to meet the dual requirements of high performance and high efficiency. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a multimodal large language model inference method based on parallel visual token scheduling. This method requires no retraining or additional modules, enables efficient pruning and information reuse of visual tags, maintains high task performance while reducing inference overhead, and is compatible with mainstream high-efficiency inference frameworks. It is suitable for practical application scenarios such as text-to-image question answering, image description generation, and visual reasoning, and has promising prospects for widespread application.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A multimodal large language model inference method based on parallel visual token scheduling includes the following steps:

[0009] S1. Obtain the input information of the model, which includes image data and text query data input by the user; use the attention weight of the category label in the visual encoder to divide the visual labels into main labels and non-main labels;

[0010] S2. Construct two parallel input streams, and input the two types of visual tags and the same text tags into the first n Transformer layers of the multimodal large language model for processing;

[0011] S3. In the shallow stage, the multimodal large language model completes the transfer of visual information to text tags through the self-attention mechanism;

[0012] S4. After the nth layer, the text tag representations in the two branches are merged, and the non-subject tags are discarded. Only the subject tags are retained to enter the remaining Transformer layers for reasoning, until the multimodal large language model finally generates and outputs the target output result; the target output result includes text-image question answer, image content description or visual logic reasoning conclusion.

[0013] Preferably, the specific process of step S1 is as follows:

[0014] S11. Visual Tag Acquisition: Extract block features from the visual encoder, and obtain the visual tag set after cross-modal projection mapping of the block features. ;

[0015] S12, Visual Tag Partitioning: Visual tag set based on category tags in the visual encoder. Attention weights for each visual tag The k visual markers with the highest attention weight scores are selected as the subset of main markers. ;in, Let L be an L-dimensional real space; L is the set of visual markers. The number of tags;

[0016] visual marker set Except for the main tag subset The remaining tags are considered as a subset of non-subject tags. .

[0017] Preferably, the specific process of step S2 is as follows:

[0018] S21. Construct two parallel input streams, represented as follows: , ,in, The input sequence for the main path; The input sequence is a non-main path; For system labeling; Text tags corresponding to text query data; For marking splicing operations; A subset of the main visual markers; This is a subset of non-subjective visual markers;

[0019] S22. The two input streams are stacked in batches and passed through the first n layers of the Transformer layer to obtain the corresponding intermediate representations: , ,in, For the front A stacked module of Transformer layers; The initial input sequence for the main path; The main path passes through The system's labeled representation after processing by the Transformer layer; The main path passes through The main visual representation after processing by the Transformer layer; The main path passes through Text markup representation after processing by the Transformer layer; The initial input sequence is the non-main path; Passing through non-main path The system's labeled representation after processing by the Transformer layer; Passing through non-main path Non-subject visual label representation after processing by the Transformer layer; Passing through non-main path Text markup representation after processing by the Transformer layer.

[0020] Preferably, the specific process of step S4 is as follows:

[0021] S41. Text tagging representation in the two branches and Weighted fusion is performed, and the fusion formula is as follows: ,in, For the merged text tags; and All are weighted coefficients;

[0022] S42. Construct the fused input sequence, represented as: ,in, A simplified input sequence that incorporates non-subjective visual information for subsequent reasoning;

[0023] S43. Starting from the (n+1)th layer, discard non-subject tags and retain only the subject tags and the fused text tags to perform subsequent Transformer layer inference until the multimodal large language model finally generates and outputs the target output results for the text query data.

[0024] Preferably, in step S41, and Both are 0.5.

[0025] By adopting the above technical solution, the present invention has the following beneficial effects:

[0026] 1. This invention achieves full transfer, fusion and reuse of semantic information from the two types of tags to text tags by processing subject tags and non-subject tags in parallel. It solves the problem of information loss caused by removing non-subject tags in existing methods, and in particular improves the model's ability to answer text-image questions that rely on non-subject information.

[0027] 2. This invention can achieve a visual tag pruning rate of up to 88.9%, significantly reduce the computational load of the Transformer self-attention mechanism (floating-point operations are reduced by 70%), and accelerate inference by 1.77 times, effectively reducing inference overhead and response latency.

[0028] 3. This invention requires no additional supervision information or training process, does not rely on heuristic rules, and adapts to different tasks and sample differences; it does not require modification of the model architecture, is compatible with mainstream efficient inference frameworks such as Flash-Attention and KV Cache, facilitates engineering implementation, and has strong generalization and compatibility.

[0029] 4. This invention has good adaptability to the LLaVA-1.5-7B model and maintains stable performance in various tasks such as text-to-image question answering, visual reasoning, and fine-grained image understanding. For example, it can still maintain an average performance of 92.45% at an 88.9% pruning rate, demonstrating stable performance and wide adaptability. Attached Figure Description

[0030] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] like Figure 1 As shown, a multimodal large language model inference method based on parallel visual token scheduling includes the following steps:

[0033] S1. Obtain the input information of the model, which includes image data and text query data input by the user; use the attention weight of the category label in the visual encoder to divide the visual labels into main labels and non-main labels;

[0034] The specific process of step S1 is as follows:

[0035] S11. Visual Tag Acquisition: Extract block features from the visual encoder, and obtain the visual tag set after cross-modal projection mapping of the block features. ;

[0036] S12, Visual Tag Partitioning: Visual tag set based on category tags in the visual encoder. Attention weights for each visual tag The k visual markers with the highest attention weight scores are selected as the subset of main markers. ;in, Let L be an L-dimensional real space; L is the set of visual markers. The number of tags;

[0037] visual marker set Except for the main tag subset The remaining tags are considered as a subset of non-subject tags. .

[0038] S2. Construct two parallel input streams, and input the two types of visual tags and the same text tags into the first n Transformer layers of the multimodal large language model for processing;

[0039] The specific process of step S2 is as follows:

[0040] S21. Construct two parallel input streams, represented as follows: , ,in, The input sequence for the main path; The input sequence is a non-main path; For system labeling; The text query data corresponds to text tags; For marking splicing operations; A subset of the main visual markers; This is a subset of non-subjective visual markers;

[0041] S22. The two input streams are stacked in batches and passed through the first n layers of the Transformer layer to obtain the corresponding intermediate representations: , ,in, For the front A stacked module of Transformer layers; The initial input sequence for the main path; The main path passes through The system's labeled representation after processing by the Transformer layer; The main path passes through The main visual representation after processing by the Transformer layer; The main path passes through Text markup representation after processing by the Transformer layer; The initial input sequence is the non-main path; Passing through non-main path The system's labeled representation after processing by the Transformer layer; Passing through non-main path Non-subject visual label representation after processing by the Transformer layer; Passing through non-main path Text markup representation after processing by the Transformer layer.

[0042] S3. In the shallow stage, the multimodal large language model completes the transfer of visual information to text tags through the self-attention mechanism;

[0043] S4. After the nth layer, the text tag representations in the two branches are fused, and non-subject tags are discarded. Only the subject tags are retained to enter the remaining Transformer layers for reasoning, until the multimodal large language model finally generates and outputs the target output result; the target output result includes text-based question-and-answer answers, image content descriptions, or visual logic reasoning conclusions;

[0044] The specific process of step S4 is as follows:

[0045] S41. Text tagging representation in the two branches and Weighted fusion is performed, and the fusion formula is as follows: ,in, For the merged text tags; and All are weighted coefficients;

[0046] S42. Construct the fused input sequence, represented as: ,in, A simplified input sequence that incorporates non-subjective visual information for subsequent reasoning;

[0047] S43. Starting from the (n+1)th layer, discard non-subject tags and retain only the subject tags and the fused text tags to perform subsequent Transformer layer inference until the multimodal large language model finally generates and outputs the target output results for the text query data.

[0048] Performance testing:

[0049] The multimodal large language model inference method based on parallel visual token scheduling of the present invention and other similar methods were tested on the LLaVA-1.5-7B model, and the test results are shown in Table 1.

[0050] Table 1: Comparison results of this method with other similar methods on the LLaVA-1.5-7B model under different pruning ratios.

[0051]

[0052] As shown in Table 1, on the LLaVA-1.5-7B model, the method of the present invention can remove up to 88.9% of the visual labels, retaining only 64 labels, while still maintaining an average performance of 92.45% (relative to the full-label baseline), while the inference speedup is 1.77 times and the floating-point operation cost is reduced by 70%.

[0053] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal large language model inference method based on parallel visual token scheduling, characterized in that, Includes the following steps: S1. Obtain the input information of the model, which includes image data and text query data input by the user; use the attention weight of the category label in the visual encoder to divide the visual labels into main labels and non-main labels; S2. Construct two parallel input streams, and input the two types of visual tags and the same text tags into the first n Transformer layers of the multimodal large language model for processing; The specific process of step S2 is as follows: S21. Construct two parallel input streams, represented as follows: , ,in, The input sequence for the main path; The input sequence is a non-main path; For system labeling; Text tags corresponding to text query data; For marking splicing operations; A subset of the main visual markers; This is a subset of non-subjective visual markers; S22. The two input streams are stacked in batches and passed through the first n layers of the Transformer layer to obtain the corresponding intermediate representations: , ,in, For the front A stacked module of Transformer layers; The initial input sequence for the main path; The main path passes through The system's labeled representation after processing by the Transformer layer; The main path passes through The main visual representation after processing by the Transformer layer; The main path passes through Text markup representation after processing by the Transformer layer; The initial input sequence is the non-main path; Passing through non-main path The system's labeled representation after processing by the Transformer layer; Passing through non-main path Non-subject visual label representation after processing by the Transformer layer; Passing through non-main path Text markup representation after processing by the Transformer layer; S3. In the shallow stage, the multimodal large language model completes the transfer of visual information to text tags through the self-attention mechanism; S4. After the nth layer, the text tag representations in the two branches are fused, and non-subject tags are discarded. Only the subject tags are retained to enter the remaining Transformer layers for reasoning, until the multimodal large language model finally generates and outputs the target output result; the target output result includes text-based question-and-answer answers, image content descriptions, or visual logic reasoning conclusions; The specific process of step S4 is as follows: S41. Text tagging representation in the two branches and Weighted fusion is performed, and the fusion formula is as follows: ,in, For the merged text tags; and All are weighted coefficients; S42. Construct the fused input sequence, represented as: ,in, A simplified input sequence that incorporates non-subjective visual information for subsequent reasoning; S43. Starting from the (n+1)th layer, discard non-subject tags and retain only the subject tags and the fused text tags to perform subsequent Transformer layer inference until the multimodal large language model finally generates and outputs the target output results for the text query data.

2. The multimodal large language model inference method based on parallel visual token scheduling as described in claim 1, characterized in that, The specific process of step S1 is as follows: S11. Visual Tag Acquisition: Extract block features from the visual encoder, and obtain the visual tag set after cross-modal projection mapping of the block features. ; S12, Visual Tag Partitioning: Visual tag set based on category tags in the visual encoder. Attention weights for each visual tag The k visual markers with the highest attention weight scores are selected as the subset of main markers. ;in, Let L be an L-dimensional real space; L is the set of visual markers. The number of tags; visual marker set Except for the main tag subset The remaining tags are considered as a subset of non-subject tags. .

3. The multimodal large language model inference method based on parallel visual token scheduling as described in claim 1, characterized in that, In step S41, and Both are 0.5.

Images