A multimodal language generation method and system guided by multi-granularity visual information

Abstract

The present invention relates to a multimodal language generation method and system guided by multi-granularity visual information. The method includes the following steps: obtaining visual input and text input, extracting text input features based on the text input, processing the visual input through a multi-granularity visual fusion module, splicing it with the text input features, and inputting it into a large language model for decoding to obtain an output response; the multi-granularity visual fusion module interpolates the visual input into low-resolution data and high-resolution data, and performs feature extraction on each to obtain low-resolution features and high-resolution features, and extracts object-level features; after fusing the low-resolution features with the high-resolution features, a fused feature is obtained, which is then spliced with the object-level features to obtain an output feature result. Compared with the existing technology, the present invention has the advantages of improving the fine-grained perception and comprehension capabilities of the multimodal large language model and improving the performance of the multimodal large language model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-modal language generation method and system guided by multi-granularity visual information. Background Art

[0002] Traditional multimodal large language models, such as LLaVA, Figure 1 As shown in the figure, it adopts the paradigm of visual encoder-projection layer-large language model. The visual input is encoded by the visual encoder, and then the features are aligned with the input of the large language model through the projection layer. Finally, it is concatenated with the text input and sent to the large language model for decoding. However, due to the significant increase in computational complexity caused by the change in the input length of the large language model, LLaVA only uses low-resolution (336×336) visual input. Later work, such as Figure 2 As shown in the figure, LLaVA-UHD improves the resolution of the visual input by splitting the high-resolution input into multiple parts, inputting them into the visual encoder separately and then splicing them together, thereby further improving the performance of the multimodal large language model.

[0003] Most existing large multimodal language models use only low-resolution inputs, resulting in poor performance in recognizing small objects and difficulties in recognizing text. Directly increasing the resolution of the visual input would significantly increase the input length, resulting in prohibitive computational costs. Furthermore, while splitting the high-resolution input into multiple inputs can alleviate this problem, the model still lacks more fine-grained information about the visual input, resulting in the model failing to perceive some objects in the scene. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies, such as directly improving the resolution of visual input, which will significantly increase the input length and bring unbearable computational costs; dividing the high-resolution input into multiple inputs, but still lacking more fine-grained information of the visual input, thus causing the model to be unable to perceive some objects in the scene. The present invention provides a multimodal language generation method and system guided by multi-granularity visual information.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A multimodal language generation method guided by multi-granularity visual information comprises the following steps:

[0007] Obtain visual input and text input, extract text input features based on the text input, process the visual input through a multi-granularity visual fusion module, concatenate it with the text input features, and input it into a large language model for decoding to obtain an output response;

[0008] The multi-granularity visual fusion module interpolates the visual input into low-resolution data and high-resolution data, and performs feature extraction respectively to obtain low-resolution features and high-resolution features, and extracts object-level features; after fusing the low-resolution features with the high-resolution features, a fused feature is obtained, which is then spliced with the object-level features to obtain the output feature results.

[0009] Furthermore, the multi-granularity visual fusion module uses a low-resolution visual encoder to extract features from low-resolution data, and uses a high-resolution visual encoder to extract features from high-resolution data.

[0010] Furthermore, the object-level feature extraction process is specifically as follows:

[0011] A labeling model is used to label the visual input by category, and the bounding box of the visual input is obtained based on the visual input and the labeled category. Based on the bounding box of the visual input and the high-resolution features, the object-level features corresponding to the bounding box are obtained by aligning the feature regions, and the obtained object-level features are spliced.

[0012] Furthermore, in the process of splicing each object-level feature, each object-level feature is globally average pooled and then spliced.

[0013] Furthermore, the process of fusing low-resolution features with high-resolution features is as follows:

[0014] The low-resolution features and the high-resolution features are respectively subjected to channel transformation by a one-dimensional convolution module; the two transformed features are spliced together and sent to the access control network to generate the weight of the high-resolution feature; the weight of the high-resolution feature is multiplied by the high-resolution feature and then added to the low-resolution feature to obtain the fused feature.

[0015] Furthermore, the visual input is a picture or a video.

[0016] Furthermore, the fusion features and object-level features are respectively converted into features suitable for a large language model through a projection layer and then concatenated.

[0017] The present invention also provides a multimodal language generation system guided by multi-granularity visual information, comprising:

[0018] Data input module, used to obtain visual input and text input;

[0019] Text embedding module, used to extract text input features based on text input;

[0020] The multi-granularity visual fusion module is used to interpolate visual input into low-resolution data and high-resolution data, and perform feature extraction to obtain low-resolution features and high-resolution features respectively, and extract object-level features; after fusing the low-resolution features with the high-resolution features, the fused features are obtained, which are then spliced with the object-level features to obtain the output feature results;

[0021] The visual text splicing module is used to splice the features output by the multi-granularity visual fusion module with the text input features;

[0022] The large language model is used to decode the splicing result output by the visual text splicing module to obtain an output response.

[0023] Furthermore, the multi-granularity visual fusion module includes:

[0024] An interpolation module, used to interpolate visual input into low-resolution data and high-resolution data;

[0025] A low-resolution visual encoder is used to extract features from low-resolution data to obtain low-resolution features;

[0026] High-resolution visual encoder, used to extract features from high-resolution data to obtain high-resolution features;

[0027] The object-level feature extraction module is used to classify the visual input using the annotation model, and obtain the bounding box of the visual input based on the visual input and the annotated category. Based on the bounding box of the visual input and the high-resolution features, the module aligns the feature regions to obtain the object-level features corresponding to the bounding box, and then splices the obtained object-level features.

[0028] The convolution gated fusion module is used to perform channel transformation on the low-resolution features and the high-resolution features respectively through the one-dimensional convolution module; the two transformed features are spliced together and sent to the gated network to generate the weight of the high-resolution feature; the weight of the high-resolution feature is multiplied by the high-resolution feature and then added to the low-resolution feature to obtain the fused feature;

[0029] The multi-granularity splicing module is used to splice the fusion features with the object-level features to obtain the output feature results.

[0030] Furthermore, the multi-granularity visual fusion module also includes a first projection layer and a second projection layer, which are respectively used to convert the fusion features and object-level features into features suitable for a large language model and then splice them.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] (1) In addition to traditional low-resolution visual input and high-resolution input, the present invention introduces object-level input into the multimodal large language model for the first time. By efficiently fusing low-resolution features, high-resolution features, and object-level features, the fine-grained perception and understanding capabilities of the multimodal large language model are improved, and the performance of the multimodal large language model is improved, especially in perception tasks.

[0033] (2) The present invention uses a lightweight convolution-gated fusion module. First, a gating network is used to generate the weight of the high-resolution feature according to the low-resolution feature and the high-resolution feature. The weight is multiplied by the high-resolution feature and then added to the low-resolution feature to obtain the fused feature. In this process, the high-resolution feature and the low-resolution feature are fused with each other, and the input length remains unchanged, which hardly brings about an increase in the amount of computation. Therefore, the overall solution greatly improves the performance of the model with a slight increase in the amount of computation.

[0034] (3) The present invention was experimentally compared with multiple large language models of different sizes, ranging from 3.8B (Billion) parameters to 34B parameters, and all achieved performance that exceeded other current advanced methods, demonstrating the generalization ability of the present invention's method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the structure of an existing low-resolution input large language model provided in the background technology of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of an existing high-resolution input large language model provided in the background technology of the present invention;

[0037] Figure 3 A schematic diagram of a structure for inputting multi-granularity visual information in a multimodal language generation method guided by multi-granularity visual information provided in an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the overall framework of a multimodal language generation method guided by multi-granularity visual information provided in an embodiment of the present invention;

[0039] Figure 5 A schematic structural diagram of a multi-granularity visual fusion module provided in an embodiment of the present invention;

[0040] Figure 6 A structural diagram of a convolution-gated fusion module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0042] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0043] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0044] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0045] Example 1

[0046] like Figure 3 As shown, this embodiment provides a multimodal language generation method guided by multi-granularity visual information, comprising the following steps:

[0047] Obtain visual input and text input, extract text input features based on the text input, process the visual input through a multi-granularity visual fusion module, concatenate it with the text input features, and input it into a large language model for decoding to obtain an output response;

[0048] The multi-granularity visual fusion module interpolates the visual input into low-resolution data and high-resolution data, and performs feature extraction respectively to obtain low-resolution features and high-resolution features, and extracts object-level features; after fusing the low-resolution features with the high-resolution features, the fused features are obtained, which are then spliced with the object-level features to obtain the output feature results.

[0049] Specifically, the overall framework of this method is as follows Figure 4 As shown in the figure, the visual input (pictures, videos, etc.) is processed by the multi-granularity visual fusion module, concatenated with the features of the text input, and then input into the large language model for decoding to obtain the output response.

[0050] The multi-granularity visual fusion module efficiently fuses three types of visual information at different granularities, including low-resolution features, high-resolution features, and object-level features, enhancing the perception and understanding capabilities of the large language model.

[0051] Specifically, the multi-granularity visual fusion module uses a low-resolution visual encoder to extract features from low-resolution data, and uses a high-resolution visual encoder to extract features from high-resolution data.

[0052] The object-level feature extraction process is as follows:

[0053] A labeling model is used to label the visual input by category, and the bounding box of the visual input is obtained based on the visual input and the labeled category. Based on the bounding box of the visual input and the high-resolution features, the object-level features corresponding to the bounding box are obtained by aligning the feature regions, and the obtained object-level features are spliced.

[0054] Preferably, in order to reduce the amount of calculation, in the process of splicing the object-level features, the object-level features are globally average pooled and then spliced.

[0055] The specific process of fusing low-resolution features with high-resolution features is:

[0056] The low-resolution features and high-resolution features are respectively subjected to channel transformation by a one-dimensional convolution module; the two transformed features are spliced together and sent to the gate control network to generate the weight of the high-resolution feature; the weight of the high-resolution feature is multiplied by the high-resolution feature and then added to the low-resolution feature to obtain the fused feature.

[0057] Optionally, the fusion features and object-level features are respectively converted into features suitable for a large language model through a projection layer and then concatenated.

[0058] In this embodiment, the multi-granularity visual fusion module is as follows: Figure 5 As shown, for the visual input, it is first interpolated into low resolution (336×336) and high resolution (768×768). For the low-resolution features, the CLIP-ViT visual encoder is used to extract features, while for the high-resolution, the CLIP-ConvNext visual encoder is used to extract features. At the same time, the annotation model RAM is used to annotate the image by category, and then the image and category are input into the open vocabulary detection model OWL-ViT2 to obtain the bounding box. According to the bounding box of the current image and the high-resolution features, the object-level features corresponding to the bounding box are obtained by feature region alignment. In order to reduce the amount of calculation, this embodiment performs global average pooling on the features corresponding to each object, and then splices the features of all objects.

[0059] This embodiment fuses low-resolution features with high-resolution features through a lightweight convolution-gating fusion module, such as Figure 6 As shown, the low-resolution and high-resolution features undergo channel transformation through two one-dimensional convolutional modules. These two transformed features are then concatenated and fed into the gated network to generate weights for the high-resolution features. This weight is multiplied by the high-resolution features and then added to the low-resolution features to create fused features. In this process, the high-resolution and low-resolution features are fused, and the input length remains unchanged, resulting in virtually no increase in computational effort. The fused features are then concatenated with object-level features and fed into the large language model for decoding.

[0060] This embodiment evaluates the above solution on multiple evaluation benchmarks and conducts multiple ablation experiments to verify the effectiveness of this method. The specific results are shown in Table 1.

[0061]

[0062] It should be noted that this method involves fusing and enhancing multiple features. Therefore, replacing other low-resolution features and high-resolution fusion modules, as well as changing the method of adding object-level features, are all variants of this algorithm and fall within the scope of protection of this method.

[0063] The above is an introduction to the method embodiment. The following further illustrates the solution of the present invention through a system embodiment.

[0064] This embodiment further provides a multimodal language generation system guided by multi-granularity visual information, including:

[0065] Data input module, used to obtain visual input and text input;

[0066] Text embedding module, used to extract text input features based on text input;

[0067] The multi-granularity visual fusion module is used to interpolate visual input into low-resolution data and high-resolution data, and perform feature extraction to obtain low-resolution features and high-resolution features respectively, and extract object-level features; after fusing the low-resolution features with the high-resolution features, the fused features are obtained, which are then spliced with the object-level features to obtain the output feature results;

[0068] The visual text splicing module is used to splice the features output by the multi-granularity visual fusion module with the text input features;

[0069] The large language model is used to decode the splicing result output by the visual text splicing module to obtain an output response.

[0070] Optional, multi-granularity visual fusion module includes:

[0071] An interpolation module, used to interpolate visual input into low-resolution data and high-resolution data;

[0072] A low-resolution visual encoder is used to extract features from low-resolution data to obtain low-resolution features;

[0073] High-resolution visual encoder, used to extract features from high-resolution data to obtain high-resolution features;

[0074] The object-level feature extraction module is used to classify the visual input using the annotation model, and obtain the bounding box of the visual input based on the visual input and the annotated category. Based on the bounding box of the visual input and the high-resolution features, the module aligns the feature regions to obtain the object-level features corresponding to the bounding box, and then splices the obtained object-level features.

[0075] The convolutional gated fusion module is used to transform the low-resolution features and high-resolution features through the one-dimensional convolution module respectively; the two transformed features are spliced together and sent to the gated network to generate the weight of the high-resolution feature; the weight of the high-resolution feature is multiplied by the high-resolution feature and then added to the low-resolution feature to obtain the fused feature;

[0076] The multi-granularity splicing module is used to splice the fusion features with the object-level features to obtain the output feature results.

[0077] Optionally, the multi-granularity visual fusion module further includes a first projection layer and a second projection layer, which are respectively used to convert the fusion features and object-level features into features suitable for a large language model and then concatenate them.

[0078] It should be noted that the specific content and beneficial effects of the system of this application can be found in the above-mentioned method embodiment and will not be repeated here.

[0079] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A multimodal language generation method guided by multi-granularity visual information, characterized in that: The following steps are involved: Obtain visual input and text input, extract text input features based on the text input, process the visual input through a multi-granularity visual fusion module, concatenate it with the text input features, and input it into a large language model for decoding to obtain an output response; The multi-granularity visual fusion module interpolates the visual input into low-resolution data and high-resolution data, and performs feature extraction on each to obtain low-resolution features and high-resolution features, and extracts object-level features; after fusing the low-resolution features with the high-resolution features, a fused feature is obtained, which is then spliced with the object-level features to obtain an output feature result; The object-level feature extraction process is specifically as follows: The annotation model is used to annotate the visual input by category, and the bounding box of the visual input is obtained based on the visual input and the annotated category. Based on the bounding box of the visual input and the high-resolution features, the object-level features corresponding to the bounding box are obtained by aligning the feature regions, and the obtained object-level features are then spliced. The specific process of fusing low-resolution features with high-resolution features is: The low-resolution features and the high-resolution features are respectively subjected to channel transformation by a one-dimensional convolution module; the two transformed features are spliced together and sent to the access control network to generate the weight of the high-resolution feature; the weight of the high-resolution feature is multiplied by the high-resolution feature and then added to the low-resolution feature to obtain the fused feature.

2. The multimodal language generation method guided by multi-granularity visual information according to claim 1, characterized in that: The multi-granularity visual fusion module uses a low-resolution visual encoder to extract features from low-resolution data, and uses a high-resolution visual encoder to extract features from high-resolution data.

3. The multimodal language generation method guided by multi-granularity visual information according to claim 1, characterized in that: In the process of splicing each object-level feature, each object-level feature is globally average pooled and then spliced.

4. The multimodal language generation method guided by multi-granularity visual information according to claim 1, characterized in that: The visual input is a picture or a video.

5. The multimodal language generation method guided by multi-granularity visual information according to claim 1, characterized in that: The fusion features and object-level features are respectively converted into features suitable for a large language model through a projection layer and then concatenated.

6. A multimodal language generation system guided by multi-granularity visual information, characterized in that: include: Data input module, used to obtain visual input and text input; Text embedding module, used to extract text input features based on text input; The multi-granularity visual fusion module is used to interpolate visual input into low-resolution data and high-resolution data, and perform feature extraction to obtain low-resolution features and high-resolution features respectively, and extract object-level features; after fusing the low-resolution features with the high-resolution features, the fused features are obtained, which are then spliced with the object-level features to obtain the output feature results; The visual text splicing module is used to splice the features output by the multi-granularity visual fusion module with the text input features; A large language model is used to decode the splicing result output by the visual text splicing module to obtain an output response; The multi-granularity visual fusion module includes: An interpolation module, used to interpolate visual input into low-resolution data and high-resolution data; A low-resolution visual encoder is used to extract features from low-resolution data to obtain low-resolution features; High-resolution visual encoder, used to extract features from high-resolution data to obtain high-resolution features; The object-level feature extraction module is used to classify the visual input using the annotation model, and obtain the bounding box of the visual input based on the visual input and the annotated category. Based on the bounding box of the visual input and the high-resolution features, the module aligns the feature regions to obtain the object-level features corresponding to the bounding box, and then splices the obtained object-level features. The convolution gated fusion module is used to perform channel transformation on the low-resolution features and the high-resolution features respectively through the one-dimensional convolution module; the two transformed features are spliced together and sent to the gated network to generate the weight of the high-resolution feature; the weight of the high-resolution feature is multiplied by the high-resolution feature and then added to the low-resolution feature to obtain the fused feature; The multi-granularity splicing module is used to splice the fusion features with the object-level features to obtain the output feature results.

7. The multimodal language generation system guided by multi-granularity visual information according to claim 6, characterized in that: The multi-granularity visual fusion module further includes a first projection layer and a second projection layer, which are respectively used to convert the fusion features and the object-level features into features suitable for a large language model and then splice them.

Images