Implementation method of high-resolution multi-modal large model architecture

Abstract

An implementation method of a high-resolution multi-modal large model architecture realizes a multi-modal large model architecture, is configured for an existing multi-modal large model and is used for processing input containing a high-resolution image, the multi-modal large model architecture introduces a local random sampling mechanism to a visual feature extraction part, and the multi-modal large model architecture is used for processing the input containing the high-resolution image. And a layered injection mechanism is adopted to inject multiple random sampling results into the LLM in a layered manner. According to the method, the visual features obtained by multiple local random sampling are divided into different layers of the LLM for injection, so that the visual features needing to be processed in each layer of the LLM are remarkably reduced, the time overhead and video memory overhead during model training and reasoning are saved on the premise that the performance of the MLLM model is ensured, and the model training and reasoning efficiency is improved. According to the local random sampling mechanism, a large number of redundant visual features are removed, the performance of the LLM model is improved, and the problem that the performance is reduced when a high-resolution image is input into the model is solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, designs computer vision technology and natural language processing technology, and is a method for implementing a high-resolution multimodal large model architecture. Background Art

[0002] As multimodal large language models (MLLMs) have shown amazing capabilities in multimodal content understanding and dialogue combining images and text, multimodal large models such as GPT-4o, LLaVA, InternVL, and Qwen-VL have emerged. These MLLMs are based on existing visual encoders and large language models (LLMs). They are first pre-trained using massive multimodal image-text data, and then supervised instruction fine-tuning is performed on high-quality multimodal instruction fine-tuning data. Relying on the powerful text and instruction understanding capabilities of large language models, and after learning on a large amount of image and text data, these models can provide powerful multimodal question-answering and dialogue functions, becoming intelligent assistants for users in their daily lives, work, and study.

[0003] In the actual application scenarios of multimodal large models, a large number of high-resolution images need to be processed, including 720P (1280×720 pixels), 1080P (1920×1280 pixels) and even up to 4K (3840x2150 pixels) resolution. How to design a multimodal large model architecture that can effectively and efficiently process high-resolution image inputs is an urgent problem to be solved in the industry. When processing high-resolution images, the existing technical solutions directly expand the image features encoded by the visual encoder in the sequence dimension and input them into the LLM. However, this results in the number of visual features far exceeding the text features, which brings significant computational efficiency and model performance issues.

[0004] The existing high-resolution multimodal large model architecture has the following main shortcomings:

[0005] (1) High computational overhead, increased training and inference costs:

[0006] The existing high-resolution MLLM architecture directly inputs a large number of image features obtained after high-resolution images are encoded by a visual encoder into the LLM, which significantly increases the length of the feature sequence that the LLM needs to process, and significantly increases the time and memory overhead during training and inference.

[0007] (2) Redundancy in visual features reduces model performance:

[0008] In the existing high-resolution MLLM architecture, the number of high-resolution image features that LLM needs to process is too large and there is serious redundancy, which causes LLM to be disturbed by a large number of redundant features during reasoning, and the accuracy and reliability of the output text response are reduced.

[0009] For example, patent application CN202410870933.X, "Efficient large-resolution image visual marker generation method for multimodal large models", discloses an efficient large-resolution image visual marker generation method for multimodal large models. First, the image is preprocessed and segmented: the original image is segmented based on the dynamic slicing method with minimum padding to obtain several image blocks. The original image is then preprocessed and then added to the set of image blocks to be processed. Feature extraction stage: A visual encoder is used to perform multi-layer visual feature extraction on each image block. The number of image features extracted by this method is proportional to the resolution of the image, and the number of image features is not compressed before being input into the LLM. This results in a significant increase in the image feature length input to the LLM when processing high-resolution images, and a significant decrease in the computational efficiency of the model. Summary of the invention

[0010] The technical problem to be solved by the present invention is that the existing multimodal large model has high computational overhead, visual features are redundant, model training and reasoning costs are high, model performance cannot meet the processing requirements of a large number of high-resolution image features, and accuracy and reliability are insufficient.

[0011] The technical solution of the present invention is: a method for implementing a high-resolution multimodal large model architecture, wherein the multimodal large model includes a visual encoder and a large language model LLM, and a multimodal large model architecture is configured on an existing multimodal large model for processing inputs containing high-resolution images. The multimodal large model architecture introduces a local random sampling mechanism into the visual feature extraction part, and adopts a hierarchical injection mechanism to hierarchically inject the results of multiple random samplings into the LLM, including:

[0012] Local random sampling: The input high-resolution image is divided into grids, and the sub-images in the grid are encoded by the visual encoder to obtain sub-image features. After the sub-image features are projected into the embedding space of the LLM, the visual features obtained by projection are spliced ​​according to the sub-image positions to obtain a high-resolution visual feature map. The high-resolution image feature map is evenly divided into local random sampling units, each unit is randomly sampled k times, and the sampling results are grouped by the number of times to obtain k groups of local random sampling results;

[0013] Layered injection: Select k layers in all Transformer layers of LLM and insert k groups of local random sampling results respectively. Configure the two hyperparameters of starting insertion layer Ls and interval T. The starting insertion layer Ls represents the first layer in LLM into which local random sampling results are inserted. The interval T represents the interval between two adjacent layers in LLM into which local random sampling results are inserted. The number of layers selected is The local random sampling result is inserted before the text features of each layer.

[0014] Furthermore, in local random sampling, the input high-resolution image is first dynamically grid matched. The grid includes H×W sub-images of size A×B. The sub-image size is equal to the input image size of the visual encoder, that is, the grid image size is HA×WB. The high-resolution image is scaled to HA×WB and then evenly divided into H×W sub-images of size A×B.

[0015] Furthermore, in local random sampling, the sub-image is encoded by a visual encoder with a downsampling rate of c. After encoding each sub-image, h×w sub-image features are obtained. All visual features are projected into the embedding space of the LLM through the visual-text connector. According to the spatial position of each sub-image in the grid, all the projected visual features are arranged and combined to obtain a high-resolution visual feature map.

[0016] Furthermore, in local random sampling, local random sampling is performed on the high-resolution image feature map:

[0017] 1) According to the deployment and application scenarios of the multimodal large model, the visual feature sampling rate is set to 1 / α, α>1, and the size of α is inversely proportional to the computational overhead of LLM processing visual features;

[0018] 2) The high-resolution visual feature map is evenly divided into local random sampling units. The number of visual features contained in each local random sampling unit is M×N, where The total number of local random sampling units is s = ((Hh) / M) × ((Ww) / N);

[0019] 3) Perform k non-repetitive random local sampling on all visual features in each local random sampling unit independently to obtain k sampling results;

[0020] 4) Take the 1st, 2nd, ..., kth sampling results in each local random sampling unit respectively, group them by number, and obtain k groups of local random sampling results that are not repeated, expressed as

[0021] Furthermore, in the layered injection, for the Ls layer, the text feature sequence obtained after the text input of LLM is processed by the text embedding layer, the first group of k groups of local random sampling results is inserted before the text feature sequence, and the sequence after the insertion of the local random sampling results is used as the input of the next Transformer layer of LLM, which is separated from the Ls layer by T layers, that is, at the Ls+T layer, the position corresponding to the local random sampling result in front of the text feature sequence is replaced by the second group of local random sampling results, until the selected k-layer Transformer layer completes the insertion of k groups of local random sampling results.

[0022] In view of the shortcomings of the prior art, the present invention proposes a new implementation method of a high-resolution MLLM architecture. The existing MLLM model can better cope with high-resolution images under this architecture. The architecture of the present invention consists of two core mechanisms: a local random sampling mechanism for high-resolution visual features, and a hierarchical injection mechanism based on multiple random sampling results. The method of the present invention significantly improves the computational efficiency and model performance of the high-resolution multimodal large model, solves the main defects in the prior art, and promotes the efficiency and intelligence of the multimodal large model in processing high-resolution image input, and has the following beneficial effects:

[0023] (1) Reduce computing overhead and save training and inference costs:

[0024] The present invention performs local random sampling on high-resolution visual features, and divides the visual features obtained by multiple sampling into different layers of LLM for injection, which significantly reduces the visual features that need to be processed by LLM in each layer. While ensuring the excellent high-resolution understanding and dialogue performance of MLLM, the computational overhead of high-resolution MLLM is greatly reduced, saving the time overhead and video memory overhead during MLLM training and inference.

[0025] (2) Reduce the redundancy of visual features and enhance model performance:

[0026] The present invention utilizes the spatial redundancy of high-resolution visual features, performs local random sampling on visual features, removes a large number of redundant visual features, and largely alleviates the interference of a large number of redundant visual features on the LLM, thereby significantly improving the performance of the LLM model and solving the problem of performance degradation when the model is fed with high-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the local random sampling mechanism in the high-resolution multimodal large model architecture implemented in the present invention.

[0028] Figure 2 A schematic diagram of a hierarchical injection mechanism based on multiple random sampling results in the high-resolution multimodal large model architecture implemented in the present invention. DETAILED DESCRIPTION

[0029] The present invention proposes a method for implementing a high-resolution multimodal large model architecture, a local random sampling mechanism for high-resolution visual features, and a hierarchical injection mechanism for LLM based on local random sampling, which utilizes the spatial redundancy of high-resolution visual features, first projects the visual features of the visual encoder into the embedding space of the LLM, then performs local random sampling on the projected visual features, and finally injects the random sampling results into the LLM. The two mechanisms work together to reduce the computational overhead of the multimodal large model for high-resolution images. By performing local random sampling on visual features, a large number of redundant visual features are removed, which greatly alleviates the interference of a large number of redundant visual features on the LLM, thereby significantly improving the performance of the LLM model. At the same time, by designing a sampling module before inputting the LLM and performing local random sampling on the visual features, the number of visual features input to the LLM is reduced, thereby reducing the computational overhead of the model. On the basis of the local random sampling mechanism, a hierarchical injection mechanism based on multiple random sampling results is proposed. The visual features obtained by multiple samplings are divided into different layers of LLM for injection, which significantly reduces the number of visual features that need to be processed at each layer of LLM. While ensuring the excellent high-resolution understanding and dialogue performance of MLLM, the computational overhead of high-resolution MLLM is greatly reduced, saving the time and memory overhead during MLLM training and inference.

[0030] Specifically, assuming that the total number of visual features is S, local random sampling obtains K sampling results, and the number of visual features contained in each sampling result is S / K. If all the sampling results are simply input into the first layer of LLM, then each layer of LLM will have to process S visual features, without reducing the computational overhead. Based on local random sampling, the present invention uses a distributed injection mechanism to input the K sampling results into K different LLM layers respectively, so that each layer of LLM only needs to process S / K visual features, reducing the computational overhead.

[0031] The implementation of the local random sampling mechanism and the layered injection mechanism of the present invention is specifically described below.

[0032] 1. Local random sampling mechanism of high-resolution visual features

[0033] The process of local random sampling mechanism of high-resolution visual features is shown as follows Figure 1 shown.

[0034] 1) The user inputs a high-resolution image, and the image is dynamically grid-matched according to the resolution and aspect ratio of the image. The present invention unifies the grid size, and the grid includes H×W sub-images of size A×B. The sub-image size is equal to the input image size of the visual encoder, that is, the grid image size is HA×WB. The high-resolution image is scaled to HA×WB to achieve dynamic matching of different high-resolution images, and then the images are evenly divided into H×W sub-images of size A×B and input into the visual encoder for encoding.

[0035] 2) Each sub-image is input into the visual encoder for encoding. The downsampling rate of the visual encoder is c, and the number of sub-image visual features is h×w, where All sub-graph visual features are projected into the embedding space of LLM through the visual-text connector.

[0036] 3) According to the spatial position of each sub-image in the grid, the visual features obtained after projecting all sub-images are rearranged and combined to obtain a high-resolution visual feature map of size Hh×Ww, which contains Hh×Ww visual features.

[0037] 4) Local random sampling of high-resolution image features:

[0038] 4.1) According to the actual deployment and application scenarios of the model, set the required sampling rate 1 / α, α>1, the size of α is inversely proportional to the computational overhead of LLM processing visual features, and can be adjusted according to actual needs;

[0039] 4.2) The high-resolution visual feature map is evenly divided into local random sampling units. The number of visual features contained in each sampling unit is M×N, where The total number of local random sampling units is s = ((Hh) / M)×((Ww) / N).

[0040] 4.3) Perform k non-repetitive random local sampling on all visual features within each local random sampling unit independently to obtain k sampling results.

[0041] 4.4) Take the 1st, 2nd, ..., kth sampling results in each local random sampling unit respectively, group them by number, and get k groups of local random sampling results that are not repeated, which can be expressed as It represents the sampling of the corresponding local random sampling unit j in the i-th group of sampling results, that is, the i-th local random sampling result of the local random sampling unit j.

[0042] 2. Layered injection mechanism based on multiple random sampling results

[0043] In order to make full use of the sampling results of the local random sampling mechanism, while ensuring the excellent high-resolution understanding and dialogue performance of MLLM, and reducing the computational overhead of high-resolution MLLM, the present invention performs multiple random samplings and adopts a layered injection mechanism to inject the results of multiple random samplings into LLM in layers. The schematic diagram of the layered injection mechanism based on multiple random sampling results is as follows: Figure 2 shown.

[0044] To obtain k groups of local random sampling results, we first need to determine the Trasnsformer layers of k LLMs, which are used as the hierarchical insertion positions of the k groups of features in the LLM. In the prior art, the number of LLM layers is determined by the size of the LLM. The larger the LLM, the more layers it has. A small model usually has 32 layers, and the largest large model can reach 126 layers. The k hierarchical insertion positions are specifically controlled by the following two hyperparameters:

[0045] The initial insertion layer, Ls, represents the first layer in LLM that inserts high-resolution visual features.

[0046] The interval T between inserted layers represents the interval between two adjacent layers in LLM where high-resolution visual features are inserted.

[0047] Given the above two hyperparameters, the k hierarchical insertion positions in LLM can be calculated

[0048] In the LLM layer corresponding to each hierarchical insertion position, the local random sampling results of the present invention are inserted at a fixed position in the sequence composed of the original text features and low-resolution visual features of the existing multimodal large model in a manner that will be spliced ​​in the sequence dimension. The insertion position is before the text features of each layer, and the inserted sequence is used as the input of the next layer of LLM. As an implementation description, for the Ls layer, the text input of LLM is processed by the text embedding layer to obtain a text feature sequence, and the first group of k groups of local random sampling results is inserted before the text feature sequence, and the sequence after the insertion of the local random sampling results is used as the input of the next layer of Transformer layer of LLM, and there is an interval of T layers from the Ls layer, that is, at the Ls+T layer, the position corresponding to the local random sampling result in front of the text feature sequence is replaced with the second group of local random sampling results, and this is done until the selected k layers of Transformer layers have all completed the injection of k groups of local random sampling results.

[0049] The multimodal large model framework including the above two mechanisms is configured to the existing multimodal large model, so that the multimodal large model can quickly and accurately process inputs containing high-resolution images.

[0050] The present invention performs local random sampling of high-resolution visual features and injects the visual features obtained by multiple samplings into different layers of LLM respectively. In the processing of LLM, there is no need to process the visual features of all high-resolution images, thereby improving the model performance and computational efficiency of the high-resolution multimodal large model, reducing the model training and inference deployment costs of enterprises, and providing enterprises with an efficient and high-performance general-purpose high-resolution multimodal large model intelligent assistant, which can be widely used in high-resolution business scenarios such as smart security and web page recognition, and improving the performance and efficiency of related intelligent algorithms.

Claims

1. A method for implementing a high-resolution multimodal large model architecture, characterized by The multimodal large model includes a visual encoder and a large language model LLM, and implements a multimodal large model architecture configured on an existing multimodal large model for processing inputs containing high-resolution images. The multimodal large model architecture introduces a local random sampling mechanism into the visual feature extraction part, and adopts a hierarchical injection mechanism to hierarchically inject the results of multiple random samplings into the LLM, including: Local random sampling: The input high-resolution image is divided into grids, and the sub-images in the grid are encoded by the visual encoder to obtain sub-image features. After the sub-image features are projected into the embedding space of the LLM, the visual features obtained by projection are spliced ​​according to the sub-image positions to obtain a high-resolution visual feature map. The high-resolution image feature map is evenly divided into local random sampling units, each unit is randomly sampled k times, and the sampling results are grouped by the number of times to obtain k groups of local random sampling results; Layered injection: Select k layers in all Transformer layers of LLM and insert k groups of local random sampling results respectively. Configure the two hyperparameters of starting insertion layer Ls and interval T. The starting insertion layer Ls represents the first layer in LLM into which local random sampling results are inserted. The interval T represents the interval between two adjacent layers in LLM into which local random sampling results are inserted. The number of layers selected is The local random sampling result is inserted before the text features of each layer.

2. The method for implementing a high-resolution multimodal large model architecture according to claim 1, characterized in that In local random sampling, the input high-resolution image is first dynamically grid matched. The grid includes H×W sub-images of size A×B. The sub-image size is equal to the input image size of the visual encoder, that is, the grid image size is HA×WB. The high-resolution image is scaled to HA×WB and then evenly divided into H×W sub-images of size A×B.

3. The method for implementing a high-resolution multimodal large model architecture according to claim 2, characterized in that In local random sampling, the sub-image is encoded by a visual encoder with a downsampling rate of c. After encoding each sub-image, h×w sub-image features are obtained. All visual features are projected into the embedding space of the LLM through the visual-text connector. According to the spatial position of each sub-image in the grid, all the projected visual features are arranged and combined to obtain a high-resolution visual feature map.

4. The method for implementing a high-resolution multimodal large model architecture according to any one of claims 1 to 3, characterized in that In local random sampling, local random sampling is performed on the high-resolution image feature map: 1) According to the deployment and application scenarios of the multimodal large model, the visual feature sampling rate is set to 1 / α, α>1, and the size of α is inversely proportional to the computational overhead of LLM processing visual features; 2) The high-resolution visual feature map is evenly divided into local random sampling units. The number of visual features contained in each local random sampling unit is M×N, where The total number of local random sampling units is s = ((Hh) / M) × ((Ww) / N); 3) Perform k non-repetitive random local sampling on all visual features in each local random sampling unit independently to obtain k sampling results; 4) Take the 1st, 2nd, ..., kth sampling results in each local random sampling unit respectively, group them by number, and obtain k groups of local random sampling results that are not repeated, expressed as 5. According to the implementation method of a high-resolution multimodal large model architecture described in claim 1, it is characterized in that in the layered injection, for the Ls layer, the text input of the LLM is processed by the text embedding layer to obtain a text feature sequence, the first group of k groups of local random sampling results is inserted before the text feature sequence, and the sequence after the insertion of the local random sampling results is used as the input of the next layer of Transformer layer of the LLM, and is separated from the Ls layer by T layers, that is, at the Ls+T layer, the position corresponding to the local random sampling result in front of the text feature sequence is replaced with the second group of local random sampling results, until the selected k layers of Transformer layers complete the insertion of k groups of local random sampling results.

Images