Vision and language fused multi-modal large model system

Abstract

The invention discloses a vision and language fused multi-modal large model system, which comprises a vision encoder fused with DINOv2 and SigLIP and used for collecting low-level spatial attributes and semantic attributes; and the multi-layer perceptron projector is used for mapping the visual features to a language embedding space and a Mama backbone model network based on a state space model. Compared with a multi-modal large language model which depends on a Transform network as a basic model, the large model system disclosed by the invention has the advantage that indexes such as reasoning speed and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal language models, and in particular to a multimodal large model system integrating vision and language. Background Art

[0002] Recently, multimodal large language models (MLLMs) have achieved impressive results in various downstream tasks, including multimodal content generation, vision-based question answering, and embodied intelligence. These advances not only highlight the versatility of MLLMs, but also pave the way for further research on more subtle and complex applications. For example, as MLLMs continue to advance, significant improvements are likely to be achieved in real-time interactions in dynamic environments, cross-modal retrieval tasks, and the seamless integration of language and vision processing in everyday technologies.

[0003] MLLMs usually rely on the well-known Transformer network as the base model for many downstream tasks. However, due to its quadratic computational complexity, the Transformer network is often inefficient, which makes it difficult to meet the needs in application scenarios that require high real-time performance and are suitable for edge deployment.

[0004] Traditional methods mainly improve the efficiency of multimodal large language models (MLLMs) by reducing model capacity or compressing the length of visual context, while usually keeping the Transformer architecture unchanged in the language model. Although this method improves efficiency, it often comes at the cost of significantly reducing model performance. Summary of the invention

[0005] The purpose of the present invention is to solve the above problems existing in the prior art and provide a multimodal system based on the Mamba language model.

[0006] The specific technical solutions adopted by the present invention are as follows:

[0007] A multimodal large model system integrating vision and language, the system comprising:

[0008] (1) An encoder module, including a visual encoder and a text encoder. The visual encoder includes a visual encoder DINOv2 and a visual encoder SigLIP. DINOv2 captures low-level spatial information in the image, and SigLip captures semantic information in the image. First, image blockification: all images are scaled and cropped into a unified format. For a given image, , where C is the number of channels of the image, H and W represent the height and width of the image respectively, and R represents the tensor space; the image is divided into Patch, where P is the side length of each patch; both the visual encoder DINOv2 and the visual encoder SigLIP take the image after the image is blocked as the input sequence, and extract the channel cascade of the two visual encoder outputs as a compact visual token ;in and Represents the feature dimensions of the output of the two visual encoders It is expressed as: ,in and They represent the feature extraction operations of DINOv2 and SigLIP on the input image, and the outputs of the two visual encoders are concatenated to form The output of the dimension; the text encoder uses GPT-NeoXTokenizer to convert the text into text tokens;

[0009] (2) The projector module uses a multi-layer perceptron (MLP) to convert the visual tokens output by the visual encoder module into Reduce the dimension to the visual token of the dimension of the next module input , the formula is , Represented as a multi-layer perceptron; the purpose of this step is to transform the dimension of the visual token into the dimension of the next module input. The output of the projector module will be sent to the next module for processing;

[0010] (3) Mamba backbone model: select the 2.8B parameter version pre-trained on the SlimPa-jama dataset or the 7B version pre-trained on the RefinedWeb dataset, and then concatenate the output of the projector module with the output of the text encoder and record it as , divided into two parts: the representation of text tokens , the representation of visual tokens , converting the input token sequence into the output target tokens sequence through autoregression , where L is the length of the output sequence, Represents the i-th element in the output sequence, and the formula is:

[0011]

[0012] in It is a text token representation. It is a visual token representation.

[0013] Furthermore, the image is scaled and cropped into a uniform format, cropped with a resolution of 384*384, and pixel values ​​are normalized.

[0014] Furthermore, the multi-layer perceptron MLP consists of two fully connected layers, and the activation function is GELU.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1) The present invention adopts the Mamba model based on the state space model, which has linear computational complexity and excels in efficient processing of long sequences, fast reasoning, and linear scalability of sequence length;

[0017] 2) A large number of experiments were conducted on three benchmark datasets to prove that the method of the present invention can significantly improve the training speed, inference speed and overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of the present invention;

[0019] Figure 2 The figure is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0021] The specific structure of the present invention is as follows:

[0022] The encoder module uses a combination of DINOv2 and SigLIP as the visual encoder, where DINOv2 is used to capture the low-level spatial attributes of the input image, and SigLIP is used to capture the semantic attributes of the input image. The feature vectors output by the two are combined as the output and sent to the projection layer for the next step of processing.

[0023] The projector module uses a multi-layer perceptron (MLP) as the projection layer to reduce the dimension of the feature vector output from the visual encoder so that the dimension of the visual feature vector is consistent with the feature vector of the text token.

[0024] The Mamba backbone model uses the Mamba backbone network as the model architecture, including short convolutional layers, SSM modules, residual connections, and RM-SNorm regularization methods.

[0025] Example

[0026] In this embodiment, a multimodal large model system based on the Mamba language model extension is proposed to solve the following problem: the multimodal large model based on the Transformer network is often inefficient, which makes it difficult to meet the requirements in application scenarios that require high real-time performance and are suitable for edge deployment.

[0027] like Figure 1 As shown, this embodiment specifically includes:

[0028] The encoder module uses a combination of DINOv2 and SigLIP as the visual encoder, where DINOv2 is used to capture the low-level spatial attributes of the input image, and SigLIP is used to capture the semantic attributes of the input image. The feature vectors output by the two are combined as the output and sent to the projection layer for the next step of processing.

[0029] It should be noted that in the encoder module, considering the input image , the visual encoder divides the image into blocks of equal size, is the size of the block. Both visual encoders take the inpainted image as input token sequence and extract the channel concatenation of the two encoder outputs as a compact visual representation :

[0030]

[0031] The projector module uses a multi-layer perceptron MLP to convert the visual token output by the visual encoder module Dimensionality reduction Visual token , the formula is , Represented as a multi-layer perceptron; the purpose of this step is to transform the dimension of the visual token into the dimension of the next module input. The output of the projector module will be sent to the next module for processing;

[0032] The Mamba backbone model uses the 2.8B parameter version pre-trained on the SlimPa-jama dataset or the 7B version pre-trained on the RefinedWeb dataset. The output of the projector module is concatenated with the output of the text encoder and recorded as , divided into two parts: the representation of text tokens , the representation of visual tokens , converting the input token sequence into the output target tokens sequence through autoregression , where L is the length of the output sequence, Represents the i-th element in the output sequence, and the formula is:

[0033]

[0034] in It is a text token representation. It is a visual token representation.

[0035] See also Figure 2 The training and experimental process of Cobra, a multimodal large model based on the Mamba language model extension, is as follows:

[0036] 1) Dataset

[0037] The following datasets are used: (1) The hybrid dataset used by LLaVA v1.5, which contains a total of 655K multi-turn visual-text hybrid dialogues including academic VQA. (2) The LVIS-Instruct-4V dataset, which contains 220K images generated by GPT-4V, which have visual alignment and context-aware internal structure. (3) The LRV-Instruct dataset, which contains 400K visual instructions covering 16 visual language tasks designed to reduce hallucinations. Overall, the entire dataset contains about 1.2 million images, corresponding multi-turn dialogue data, and plain text dialogue data.

[0038] 2) Metrics

[0039] Experiments are conducted on nine different benchmarks. They include: (1) four open-ended visual question answering tasks, namely VQA-v2, GQA, VizWiz, and TextVQA. (2) two closed-ended visual question answering tasks, namely VSR and POPE. (3) three visual pointing tasks, namely RefCOCO, RefCOCO+, and RefCOCOg. VQA-v2 (Goyal et al., 2017) is used to evaluate the model's understanding and reasoning ability of images and questions. GQA (Hudson and Manning, 2019) focuses on spatial understanding and multi-dimensional reasoning and is tested on real-world images. VizWiz (Gurari et al., 2018) is similar to VQA-v2, but adds some unanswered questions to test the model's ability to respond. TextVQA (Singh et al., 2019) focuses on reasoning from textual information in images. VSR (Liu, Emerson, and Collier, 2023) consists of a series of yes / no questions that examine the model's ability to judge spatial relations in complex scenes, which poses a challenge to multimodal large language models (MLLMs). POPE (Li et al., 2023b) consists of specific yes / no questions to evaluate the model's tendency to generate hallucinated content. RefCOCO mainly handles short descriptions with spatial anchors, RefCOCO+ relies on appearance-based descriptions, and RefCOCOg emphasizes the use of lengthy and rich descriptions (Kazemzadeh et al., 2014; Yu et al., 2016).

[0040] 3) Baseline Method

[0041] Cobra is compared with a variety of algorithms of different scales, including: (1) large-scale multimodal large language models (MLLMs): OpenFlamingo (Awadalla et al., 2023), BLIP-2 (Li et al., 2023a), MiniGPT-4 (Zhu et al., 2023), InstructBLIP (Dai et al., 2023), Shikra (Chen et al., 2023a), IDEFICS (Laurençon et al., 2023), Qwen-VL (Bai et al., 2023), and LLaVA v1.5 (Liu et al., 2023b). (2) small-scale MLLMs: MoE-LLaVA (Lin et al., 2024), LLaVA-Phi (Zhu et al., 2024), and MobileVLM v2 (Chu et al., 2024). Cobra leads the way in all the metrics and inference speed mentioned above.

[0042] 4) Implementation details

[0043] The training process included multimodal instruction tuning, during which the multimodal projector and the Mamba Large Language Model (LLM) were fine-tuned. The model was trained using 8 NVIDIA A100 80GB GPUs. A variety of open source model weights were selected, including Mamba with 2.8 billion and 7 billion parameters as the LLM backbone of the model.

[0044] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A multimodal large model system integrating vision and language, characterized in that: The system includes: (1) An encoder module, including a visual encoder and a text encoder. The visual encoder includes a visual encoder DINOv2 and a visual encoder SigLIP. DINOv2 captures low-level spatial information in the image, and SigLip captures semantic information in the image. First, image blockification: all images are scaled and cropped into a unified format. For a given image, , where C is the number of channels of the image, H and W represent the height and width of the image respectively, and R represents the tensor space; the image is divided into Patch, where P is the side length of each patch; both the visual encoder DINOv2 and the visual encoder SigLIP take the image after the image is blocked as the input sequence, and extract the channel cascade of the two visual encoder outputs as a compact visual token ;in and Represents the feature dimensions of the output of the two visual encoders It is expressed as: ,in and They represent the feature extraction operations of DINOv2 and SigLIP on the input image, and the outputs of the two visual encoders are concatenated to form The output of the dimension; the text encoder uses GPT-NeoXTokenizer to convert the text into text tokens; (2) The projector module uses a multi-layer perceptron (MLP) to convert the visual tokens output by the visual encoder module into Reduce the dimension to the visual token of the dimension of the next module input , the formula is , Represented as a multi-layer perceptron; (3) Mamba backbone model: select the 2.8B parameter version pre-trained on the SlimPa-jama dataset or the 7B version pre-trained on the RefinedWeb dataset, and then concatenate the output of the projector module with the output of the text encoder and record it as , divided into two parts: the representation of text tokens , the representation of visual tokens , converting the input token sequence into the output target tokens sequence through autoregression , where L is the length of the output sequence, Represents the i-th element in the output sequence, and the formula is: ， in It is a text token representation. It is a visual token representation.

2. The multimodal large model system according to claim 1, characterized in that: The image is scaled and cropped into a uniform format, cropped with a resolution of 384*384, and pixel values ​​are normalized.

3. The multimodal large model system according to claim 1, characterized in that: The multi-layer perceptron MLP consists of two fully connected layers, and the activation function is GELU.

Images