Multi-granularity data set construction method and device for multi-modal large model and storage medium

Abstract

The invention belongs to the field of multi-modal large models, and particularly relates to a multi-granularity data set construction method and device for a multi-modal large model and a storage medium. The objective of the invention is to solve the problem that an existing multi-modal large model is poor in image content understanding capability. The invention provides a multi-granularity data set construction method oriented to a multi-modal large model. The method comprises the following steps: collecting a data set; processing all the images in the data set to obtain preprocessed images; annotating the preprocessed fine-grained object; coarse-grained subtitles of all the images in the data set are annotated; describing fine-grained object category labels; and finally, constructing a multi-granularity data set of the multi-modal large model. According to the multi-granularity data set of the multi-modal large model, the problems that an existing multi-modal large model training set only uses coarse-granularity annotation and neglects fine-granularity annotation, so that the model does not deeply understand concepts, and the multi-modal task performance of the model is limited are solved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal large models, and in particular relates to a method, device and storage medium for constructing a multi-granularity data set for a multimodal large model. Background Art

[0002] As the capabilities and accessibility of large language models (LLMs) continue to improve, researchers have integrated LLMs with visual modules to build powerful multimodal large models (MLLMs), and have made significant progress in tasks such as visual question answering, image caption generation, and image generation. One type of work focuses on multimodal understanding, aligning LLMs with visual modules through additional cross-attention layers, QueryTransformer, or projection layers. Another type of research focuses on multimodal generation, aligning LLMs with visual modules by learning visual tokens or visual embeddings. Recent work (EMU, SEED-LLaMA, LaVIT, etc.) has further proposed a multimodal general model that can perform multimodal understanding and generation simultaneously.

[0003] However, existing MLLMs usually ignore the importance of concepts and only utilize large-scale coarse-grained image captions, which limits the alignment of vision and language and causes the model to have a superficial understanding of concepts in images and texts.

[0004] In traditional visual-language models (VLMs), many works recognize the importance of concepts in visual-language learning. In order to better utilize concepts and improve the alignment of vision and language, VLMs combine fine-grained concept annotations with coarse-grained image captions, such as region features, object labels, label descriptions, region descriptions, and object regions. Oscar appends fine-grained object labels detected from images to coarse-grained image captions to simplify the semantic alignment between vision and language. K-Lite uses WordNet and Wiktionary to provide fine-grained label descriptions for each object label, thereby supplementing relevant knowledge to help understand these concepts. X-VLM proposes to align visual concepts (images and object regions) with coarse-grained image captions and fine-grained object labels and object region descriptions at multiple granularities. However, most of these VLMs need to add additional components and loss functions for fine-grained annotations, and can only optimize the understanding ability of the model at different granularities. In addition, these VLMs only use certain types of fine-grained annotations, and do not fully combine coarse-grained and multiple fine-grained annotations, nor explore their effects in the field of image generation.

[0005] In summary, existing large multimodal models have the problem of poor ability to understand image content. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the existing multimodal large model has poor ability to understand image content. A multi-granularity dataset construction method for a multimodal large model is proposed, including:

[0007] Step 1: Collect data sets;

[0008] The dataset includes images, coarse-grained caption annotations of images, and fine-grained object annotations of images.

[0009] The fine-grained object annotations of the image include fine-grained object bounding box coordinates and fine-grained object category labels;

[0010] In this work, we focus on concrete concepts (concepts can be classified into concrete and abstract concepts based on whether they can be perceived by the five senses), specifically objects, object attributes, and relations between objects. These concepts are fundamental concepts in vision-language learning and are widely annotated in object detection datasets.

[0011] We collected four public large-scale manually annotated object detection datasets, including Open Images, Objects365, V3Det, ​​and Visual Genome. The images in these datasets are real-world images uploaded by users to the Flickr website and collected by the dataset providers.

[0012] The real-world images collected by the dataset providers usually show complex scenes containing multiple objects, and are accompanied by fine-grained category labels and object bounding boxes. Compared with the widely used coarse-grained image caption datasets, the fine-grained concept annotations provided by these datasets can help MLLMs localize and learn concepts in images.

[0013] Step 2: preprocess all images in the data set of step 1 to obtain preprocessed images;

[0014] Preprocessing the fine-grained object annotations in the data set in step 1 to obtain preprocessed fine-grained object annotations;

[0015] Step 3: Generate coarse-grained caption annotations for the images in the data set without coarse-grained caption annotations in step 1; add the generated coarse-grained caption annotations to the coarse-grained caption annotations in the data set to obtain the coarse-grained caption annotations for all images in the data set;

[0016] Step 4: Generate a fine-grained object category label description for the fine-grained object category label in the data set of step 1;

[0017] Step 5: Construct a multi-granular dataset of a multimodal large model based on the preprocessed images obtained in step 2, the preprocessed fine-grained object annotations, the coarse-grained image caption annotations of all images obtained in step 3, and the fine-grained object category label descriptions obtained in step 4.

[0018] A computer storage medium stores at least one instruction, which is loaded and executed by a processor to implement a multi-granularity data set construction method for a multi-modal large model.

[0019] A multi-granularity dataset construction device for a multi-modal large model, the device comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement a multi-granularity dataset construction method for a multi-modal large model.

[0020] The beneficial effects of the present invention are:

[0021] It solves the problem that the existing multimodal large model training sets only use coarse-grained annotations and ignore fine-grained annotations, which leads to the model's insufficient understanding of concepts and limits the model's multimodal task performance.

[0022] Different from the widely used image caption data paradigm, MGC is an image-text interleaved document with multi-granularity concept annotations, including both textual forms (captions, labels, and label descriptions) and visual forms (images and object regions). It can help MLLMs learn concepts better and use concepts to align vision and language at multiple granularities. In summary, MGC contains 3.5 million unique images, 344,000 unique category label description pairs, and 23.9 million unique object regions. It can help us explore the potential of MGC as a new data paradigm for MLLMs, enhance the conceptual understanding and generation capabilities of MLLMs, thereby improving the alignment of vision and language and achieving performance improvements on various multimodal downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of a multi-granularity data concentration template for a multi-modal large model of the present invention. DETAILED DESCRIPTION

[0024] Specific implementation method 1: Combination Figure 1 The invention is described, comprising:

[0025] Step 1: Collect data sets;

[0026] The dataset includes images, coarse-grained caption annotations of images, and fine-grained object annotations of images.

[0027] The fine-grained object annotations of the image include fine-grained object bounding box coordinates and fine-grained object category labels;

[0028] The coarse-grained caption annotation of the image refers to the image title, which is similar to an image of a woman holding a child;

[0029] The fine-grained object refers to an object corresponding to a certain thing in the image; for example, the sunglasses worn by the woman in the picture,

[0030] In this work, we focus on concrete concepts (concepts can be classified into concrete and abstract concepts based on whether they can be perceived by the five senses), specifically objects, object attributes, and relations between objects. These concepts are fundamental concepts in vision-language learning and are widely annotated in object detection datasets.

[0031] We collected four public large-scale manually annotated object detection datasets, including Open Images, Objects365, V3Det, ​​and Visual Genome. The images in these datasets are real-world images uploaded by users to the Flickr website and collected by the dataset providers.

[0032] The real-world images collected by the dataset providers usually show complex scenes containing multiple objects, and are accompanied by fine-grained category labels and object bounding boxes. Compared with the widely used coarse-grained image caption datasets, the fine-grained concept annotations provided by these datasets can help MLLMs localize and learn concepts in images.

[0033] Step 2: preprocess all images in the data set of step 1 to obtain preprocessed images;

[0034] Preprocessing the fine-grained object annotations in the data set in step 1 to obtain preprocessed fine-grained object annotations;

[0035] Step 3: Generate coarse-grained caption annotations for the images in the data set without coarse-grained caption annotations in step 1; add the generated coarse-grained caption annotations to the coarse-grained caption annotations in the data set to obtain the coarse-grained caption annotations for all images in the data set;

[0036] Step 4: Generate a fine-grained object category label description for the fine-grained object category label in the data set of step 1;

[0037] Step 5: Construct a multi-granularity dataset of the multimodal large model based on the preprocessed images obtained in step 2, the preprocessed fine-grained object annotations, the coarse-grained image caption annotations of all images obtained in step 3, and the fine-grained object category label descriptions obtained in step 4. The multi-granularity dataset of the multimodal large model is used for training the multimodal large model.

[0038] It can help us explore the potential of multi-granularity datasets of multimodal large models as a new data paradigm for MLLMs, enhance the conceptual understanding and generation capabilities of MLLMs, thereby improving the alignment of vision and language, and achieving performance improvements on various multimodal downstream tasks. Modal downstream tasks include image-text matching (such as correspondence between images and descriptions), visual question answering (answering questions on images), image generation (generating images based on text descriptions), video understanding (analyzing video content), multimodal sentiment analysis (combining text and visual information to analyze sentiment), and cross-modal retrieval (retrieval of content in one modality based on information in another modality).

[0039] Specific implementation method 2: The difference between this implementation method and specific implementation method 1 is that:

[0040] In the step 2, the image in the data set of step 1 is preprocessed to obtain the preprocessed image; the fine-grained object annotations in the data set of step 1 are preprocessed to obtain the preprocessed fine-grained object annotations; the specific process is:

[0041] Step 21: Preprocess the image in the data set of step 1 to obtain the preprocessed image.

[0042] Step 22: According to the fine-grained object annotations in the data set of step 1, obtain the preprocessed fine-grained object annotations;

[0043] The other steps and parameters are the same as those in the first specific implementation.

[0044] Specific implementation method three: This implementation method is different from the specific implementation method one in that:

[0045] In step 21, the image in the data set of step 1 is preprocessed to obtain a preprocessed image. The specific process is as follows:

[0046] Step 2: Delete the images whose short side is less than X pixels in the data set of step 1.

[0047] Images with aspect ratios greater than Y, images with invalid annotations, and images with low-frequency category labels; obtain a set of deleted images; X is a positive integer, preferably 224, and Y is a positive number, preferably 3

[0048] The low-frequency category labels are fine-grained object category labels that appear less than 3 times.

[0049] Step 212: adjusting all images in the deleted image set to a fixed X×X pixel resolution to obtain a preprocessed image; the preprocessed image is a square image;

[0050] The preferred method of the image adjustment method is to use the Image.reszie function of the Pillow library, whose corresponding default algorithm is the bicubic resampling algorithm, to adjust the images in the deleted image set to a fixed X×X pixel resolution, and other adjustment methods with the same function can be used;

[0051] The other steps and parameters are the same as those in the first to second embodiments.

[0052] Specific implementation method 4: This implementation method is different from specific implementation methods 1 to 4 in that:

[0053] The step 22 obtains the preprocessed fine-grained object annotations according to the fine-grained object annotations in the step 1 data set; the specific process is: the specific process of obtaining a preprocessed fine-grained object annotation for one of the fine-grained object annotations in the step 1 data set is:

[0054] Step 221: According to the bounding box coordinates in the fine-grained object annotation in the step 1 dataset, obtain the original object region R in the original image in the step 1 dataset.

[0055] Taking the center of the original object region R as the center of the square region S, a square region S including the original object region R is cut out from the original image in the data set of step 1;

[0056] The original object region R is obtained according to the bounding box coordinates in the fine-grained object annotation. For example, if the bounding box coordinates are (2, 2) (0, 0) (2, 0) (0, 2), then the center of the original object region R is (1, 1).

[0057] Step 222: Integrate the fine-grained object category label of the original object region R with other fine-grained object category labels contained in the square region S to obtain an integrated annotation, and update the integrated annotation to the fine-grained object category label of the square region S;

[0058] Step 223: Convert the square area S into an image representation of the square area S through a visual encoder, and then convert the image representation of the square area S into discrete visual tokens of the square area S through a visual tokenizer.

[0059] The fine-grained object category label and location description of the square area S in text form are converted into discrete text tokens of the square area S through a textual tokenizer;

[0060] Visual encoders commonly used by people in this field include ResNet, ViT (Vision Transformer), and SwinTransformer.

[0061] Visual word segmentation includes image segmentation models such as Mask R-CNN

[0062] Step 224: Combine the visual tokens and text tokens of the square area S into a preprocessed fine-grained object annotation of the square area S.

[0063] The fine-grained object annotations we preprocessed are originally images and texts. We will convert them into visual tokens and text tokens respectively to facilitate direct use during model training.

[0064] The other steps and parameters are the same as those in the first to third embodiments.

[0065] Specific implementation method 5: This implementation method is different from specific implementation methods 1 to 4 in that:

[0066] In step 3, a coarse-grained image caption annotation is generated for the image without coarse-grained image caption annotation in the data set of step 1; the specific process is:

[0067] For an image A in the data set in step 1 that does not have a coarse-grained image caption annotation, a coarse-grained image caption annotation is generated. The specific process includes:

[0068] Step 31: Input image A into the BLIP-2 model and output a generated caption annotation;

[0069] Step 32: Input image A and the subtitles generated in step 31 into the CLIP model, and output the score of the generated subtitles;

[0070] When the score is lower than the threshold, return to step 3 to regenerate the coarse-grained image caption annotation;

[0071] When the score is higher than the threshold, the caption annotation is generated as the coarse-grained image caption annotation of image A;

[0072] When the number of regenerations is higher than the set number of generation, the generated caption annotation with the highest score is used as the coarse-grained image caption annotation of image A; (for example, a maximum of 10 generation times is set, and the best score is taken)

[0073] The score represents the relationship score between the input image and the generated caption;

[0074] Since only some images are annotated with coarse-grained image captions, we refer to LAION-COCO and automatically generate captions for all images with the help of BLIP-2 and CLIP models.

[0075] BLIP-2 and CLIP are two deep learning models for image and text processing. CLIP (Contrastive Language-Image Pre-training) was developed by OpenAI and can embed images and text into a shared representation space, allowing the model to understand image content through language and perform retrieval for a variety of visual and language tasks.

[0076] In addition, although fine-grained attribute and relation labels are only available in some images, after manually inspecting the annotations generated by the external model, we decided to use only the human annotations provided in the original dataset to avoid introducing additional noise.

[0077] Label descriptions are the corresponding conceptual descriptions of visual concepts in images, conveying the understanding of concepts through visually observed details and related knowledge. Inspired by the success of previous works, these works introduced label descriptions through WordNet and LLMs to enhance the understanding of concepts.

[0078] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0079] Specific implementation method 6: The difference between this implementation method and specific implementation methods 1 to 5 is that in step 4, a fine-grained object category label description is generated for the fine-grained object category label in the data set of step 1; the specific process is:

[0080] Step 41: Use GPT-4 to generate an initial fine-grained object category label description for the fine-grained object category label in the step 1 data set;

[0081] Step 42: Use WordNet to check and modify the generated initial fine-grained object category description; obtain the fine-grained object category label description;

[0082] The fine-grained object category label includes: fine-grained objects, fine-grained object attributes and the relationship between fine-grained objects;

[0083] We design hint templates for each category label and carefully provide human-annotated examples.

[0084] In addition, we checked the category labels with the help of WordNet and updated the polysemous labels based on the specific data samples to better distinguish them, such as changing "batter" to "batter (ballplayer)" or "batter (cooking)".

[0085] Different from the widely used image caption data paradigm, the present invention is an image-text interleaved document with multi-granularity concept annotations, including both textual forms (captions, labels, and label descriptions) and visual forms (images and object regions). It can help MLLMs learn concepts better and use concepts to align vision and language at multiple granularities. In summary, MGC contains 3.5 million unique images, 344,000 unique category label description pairs, and 23.9 million unique object regions. It can help us explore the potential of MGC as a new data paradigm for MLLMs, enhance the conceptual understanding and generation capabilities of MLLMs, thereby improving the alignment of vision and language, and achieving performance improvements on various multimodal downstream tasks. ; The other steps and parameters are the same as those of the specific implementation methods one to five.

[0086] Specific implementation method seven: This implementation method is a computer storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the method for constructing a multi-granularity dataset for a multimodal large model.

[0087] It should be understood that the instructions include computer program products, software or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system, or other electronic device. Computer storage media may include readable media on which instructions are stored, which may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions.

[0088] Specific implementation eight: This implementation is a multi-granularity data set construction device for a multi-modal large model, the device includes a processor and a memory, it should be understood that including any device including a processor and a memory described in the present invention, the device may also include other units and modules that perform display, interaction, processing, control, etc. and other functions through signals or instructions;

[0089] At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the method for constructing a multi-granularity data set for a multi-modal large model.

[0090] Those skilled in the art will appreciate that at least one instruction stored is a computer program product corresponding to the method or system. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, systems, and computer program products according to the embodiments of the present application, and may also be used for corresponding devices. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] The above only describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above specific implementation methods. Although the present invention has been disclosed as above in the preferred embodiments, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments with equivalent changes without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for constructing a multi-granularity dataset for a multimodal large model, characterized in that: The following steps are involved: Step 1: Collect data sets; The dataset includes images, coarse-grained caption annotations of images, and fine-grained object annotations of images. The fine-grained object annotations of the image include fine-grained object bounding box coordinates and fine-grained object category labels; Step 2: preprocess all images in the data set of step 1 to obtain preprocessed images; Preprocessing the fine-grained object annotations in the data set in step 1 to obtain preprocessed fine-grained object annotations; Step 3: Generate coarse-grained caption annotations for the images without coarse-grained caption annotations in the data set of step 1; Add the generated coarse-grained caption annotations to the coarse-grained caption annotations in the dataset to obtain the coarse-grained caption annotations for all images in the dataset; Step 4: Generate a fine-grained object category label description for the fine-grained object category label in the data set of step 1; Step 5: Construct a multi-granular dataset of a multimodal large model based on the preprocessed images obtained in step 2, the preprocessed fine-grained object annotations, the coarse-grained image caption annotations of all images obtained in step 3, and the fine-grained object category label descriptions obtained in step 4.

2. The method for constructing a multi-granularity dataset for a multi-modal large model according to claim 1, characterized in that: In the step 2, the image in the data set of step 1 is preprocessed to obtain the preprocessed image; the fine-grained object annotations in the data set of step 1 are preprocessed to obtain the preprocessed fine-grained object annotations; the specific process is: Step 21: Preprocess the image in the data set of step 1 to obtain the preprocessed image. Step 22: According to the fine-grained object annotations in the data set of step 1, obtain the preprocessed fine-grained object annotations;.

3. The method for constructing a multi-granularity dataset for a multi-modal large model according to claim 2, characterized in that: In step 21, the image in the data set of step 1 is preprocessed to obtain a preprocessed image. The specific process is as follows: Step 2: Delete the images in the data set of step 1 whose short sides are less than X pixels, whose aspect ratio is greater than Y, images with invalid annotations, and images with low-frequency category labels; obtain the deleted image set; X is a positive integer, Y is a positive number, The low-frequency category labels are fine-grained object category labels that appear less than 3 times. Step 212: Adjust all images in the deleted image set to a fixed X×X pixel resolution to obtain a preprocessed image; the preprocessed image is a square image.

4. The method for constructing a multi-granularity dataset for a multi-modal large model according to claim 3, characterized in that: The step 22 obtains the preprocessed fine-grained object annotations according to the fine-grained object annotations in the data set of step 1; wherein the specific process of obtaining a preprocessed fine-grained object annotation for one of the fine-grained object annotations in the data set of step 1 is: Step 221: According to the bounding box coordinates in the fine-grained object annotation in the step 1 dataset, obtain the original object region R in the original image in the step 1 dataset. Taking the center of the original object region R as the center of the square region S, a square region S including the original object region R is cut out from the original image in the data set of step 1; Step 222: Integrate the fine-grained object category label of the original object region R with other fine-grained object category labels contained in the square region S to obtain an integrated annotation, and update the integrated annotation to the fine-grained object category label of the square region S; Step 223: Convert the square area S into an image representation of the square area S through the visual encoder, and then convert the image representation of the square area S into a visual token of the square area S through the visual tokenizer. The fine-grained object category label and location description of the square area S are converted into text tokens of the square area S through a text tokenizer; Step 224: Combine the visual tokens and text tokens of the square area S into a preprocessed fine-grained object annotation of the square area S.

5. The method for constructing a multi-granularity dataset for a multi-modal large model according to claim 4, characterized in that: In step 3, a coarse-grained image caption annotation is generated for the image without coarse-grained image caption annotation in the data set of step 1; the specific process is: For an image A in the data set in step 1 that does not have a coarse-grained image caption annotation, a coarse-grained image caption annotation is generated. The specific process includes: Step 31: Input image A into the BLIP-2 model and output a generated caption annotation; Step 32: Input image A and the subtitles generated in step 31 into the CLIP model, and output the score of the generated subtitles; When the score is lower than the threshold, return to step 3 to regenerate the coarse-grained image caption annotation; When the score is above the threshold, a caption annotation is generated as the coarse-grained image caption annotation for image A.

6. The method for constructing a multi-granularity dataset for a multi-modal large model according to claim 5, characterized in that: In step 4, a fine-grained object category label description is generated for the fine-grained object category label in the data set of step 1; the specific process is: Step 41: Use GPT-4 to generate an initial fine-grained object category label description for the fine-grained object category label in the step 1 data set; Step 42: Use WordNet to check and modify the generated initial fine-grained object category description; obtain the fine-grained object category label description.

7. A computer storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement a multi-granularity dataset construction method for a multimodal large model as described in any one of claims 1 to 6.

8. A multi-granularity dataset construction device for a multi-modal large model, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement a multi-granularity dataset construction method for a multimodal large model as described in any one of claims 1 to 6.

Images