Multimodal interleaved image-text generative model based on dynamic feature synchronizer

Abstract

The present disclosure relates to the technical field of multimodal learning, and particularly relates to a multimodal interleaved image-text generative model based on a dynamic feature synchronizer. An image encoder extracts multi-resolution multi-scale feature maps from input images of interleaved image-text data; and a dynamic feature synchronizer in a multimodal large language model acquires fine-grained information, so as to determine output feature data corresponding to the interleaved image-text data and then generate a target image and / or target text associated with the interleaved image-text data.

Description

A multimodal image-text interleaving generation model based on dynamic feature synchronizers

[0001] Cross-references to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 202410798605.3, filed by Tsinghua University on June 20, 2024, entitled "A Multimodal Image-Text Interleaving Generation Model Based on Dynamic Feature Synchronizer". Technical Field

[0003] This disclosure relates to the field of multimodal learning technology, and in particular to a multimodal image-text interleaving generation model based on a dynamic feature synchronizer. Background Technology

[0004] In recent years, with the rapid development of Large Language Models (LLM) and multimodal learning, applying LLM to process image and text data has become a research hotspot. In particular, for interleaved image and text data (such as news blogs, which contain multiple images interspersed within text) that are prevalent on the Internet, building a multimodal model that can simultaneously understand and generate such data has significant research value and application potential.

[0005] Compared to previously extensively studied image-text paired data, image-text interleaved data not only covers a wider range of scenarios but also exhibits longer and more complex multimodal structures. Therefore, developing generative models capable of handling such image-text interleaved sequences can extend and unify previous multimodal models, integrating tasks such as text-to-image generation and visual question answering, thus becoming a challenging new research topic.

[0006] However, no reasonable and effective technical solution has yet been provided for the technical problem of how to efficiently integrate detailed information from multiple multi-resolution images in a multimodal large language model.

[0007] Public content

[0008] In view of this, this disclosure proposes a multimodal image-text interleaving generation model based on a dynamic feature synchronizer.

[0009] According to one aspect of this disclosure, a multimodal image-text interleaving generation model based on a dynamic feature synchronizer is provided, the multimodal image-text interleaving generation model including an image encoder and a multimodal large language model;

[0010] The image encoder is used to extract multi-resolution multi-scale feature maps from the input image of the image-text interleaved data, which includes interleaved input images and input text.

[0011] The dynamic feature synchronizer in the multimodal large language model is used to obtain fine-grained information from the multi-resolution multi-scale feature map. The fine-grained information indicates the visual detail features of the multi-resolution multi-scale feature map, so that the multimodal large language model determines the output feature data corresponding to the image-text interleaved data based on the fine-grained information. The output feature data is used to generate target images and / or target text associated with the image-text interleaved data.

[0012] In one possible implementation, the dynamic feature synchronizer in the multimodal large language model is located between the self-attention layer and the feedforward network layer. The input parameters of the dynamic feature synchronizer include the hidden state features output by the self-attention module, and the output parameters of the dynamic feature synchronizer include state features with the same shape as the input parameters, obtained after being fused with the image features mentioned above.

[0013] In another possible implementation, the dynamic feature synchronizer in the multimodal large language model is further used to extract the fine-grained information from the multi-resolution multi-scale feature map based on the sampling point coordinates and corresponding attention weights in the multi-resolution multi-scale feature map. The fine-grained information includes the state features corresponding to each of the multiple sampling points.

[0014] In another possible implementation

[0015] The image encoder is also used to extract visual feature vectors from the input image of the image-text interleaving data;

[0016] The multimodal large language model is further used to extract text feature vectors from the input text of the image-text interleaved data; determine the input feature data corresponding to the image-text interleaved data based on the visual feature vectors and the text feature vectors; and determine the corresponding output feature data based on the input feature data and the fine-grained information.

[0017] In another possible implementation, the multimodal image-text interleaving generation model further includes: an image decoder;

[0018] The image decoder is used to generate the target image corresponding to the image-text interleaving data based on the output feature data and the multi-resolution multi-scale feature map.

[0019] In another possible implementation, the image decoder includes a dynamic feature synchronizer for acquiring the fine-grained information from the multi-resolution, multi-scale feature map, so that the image decoder performs image decoding based on the fine-grained information.

[0020] In another possible implementation, the multimodal text-image interleaving generation model further includes: a text feature classifier;

[0021] The text feature classifier is used to generate the target text corresponding to the image-text interleaving data based on the output feature data.

[0022] According to another aspect of this disclosure, a multimodal image-text interleaving generation method based on a dynamic feature synchronizer is provided, the method comprising:

[0023] Acquire image-text interleaved data, which includes interleaved input images and input text;

[0024] Based on the image-text interleaved data, the corresponding output feature data is obtained by using a multimodal large language model. The dynamic feature synchronizer in the multimodal large language model is used to obtain fine-grained information from the multi-resolution multi-scale feature map of the input image. The fine-grained information indicates the visual detail features of the multi-resolution multi-scale feature map.

[0025] Based on the output feature data, a target image and / or target text associated with the image-text interleaving data are generated.

[0026] In one possible implementation, the dynamic feature synchronizer in the multimodal large language model is located between the self-attention layer and the feedforward network layer. The input parameters of the dynamic feature synchronizer include the latent space state features output by the self-attention module, and the output parameters of the dynamic feature synchronizer include state features with the same shape as the input parameters, obtained after being fused with the image features mentioned above.

[0027] In another possible implementation, the dynamic feature synchronizer in the multimodal large language model is further used to extract the fine-grained information from the multi-resolution multi-scale feature map based on the sampling point coordinates and corresponding attention weights in the multi-resolution multi-scale feature map. The fine-grained information includes the state features corresponding to each of the multiple sampling points.

[0028] In another possible implementation, the step of obtaining corresponding output feature data based on the interleaved text and image data using a multimodal large language model includes:

[0029] Visual feature vectors are extracted from the input image of the image-text interleaved data, and text feature vectors are extracted from the input text of the image-text interleaved data;

[0030] Based on the visual feature vector and the text feature vector, determine the input feature data corresponding to the image-text interleaving data;

[0031] Based on the input feature data, the corresponding output feature data is determined through the multimodal large language model.

[0032] In another possible implementation, generating the target image and / or target text associated with the image-text interleaving data based on the output feature data includes:

[0033] Based on the output feature data and the multi-resolution, multi-scale feature map, the target image corresponding to the image-text interleaving data is generated by the image decoder.

[0034] In another possible implementation, the image decoder includes a dynamic feature synchronizer for acquiring the fine-grained information from the multi-resolution, multi-scale feature map, so that the image decoder performs image decoding based on the fine-grained information.

[0035] In another possible implementation, generating the target image and / or target text associated with the image-text interleaving data based on the output feature data includes:

[0036] Based on the output feature data, the target text corresponding to the image-text interleaving data is generated by a text feature classifier.

[0037] According to another aspect of this disclosure, a multimodal image-text interleaving generation apparatus based on a dynamic feature synchronizer is provided, the apparatus comprising:

[0038] processor;

[0039] Memory used to store processor-executable instructions;

[0040] The processor is configured to implement the above method when executing instructions stored in the memory.

[0041] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, characterized in that the computer program instructions, when executed by a processor, implement the method described above.

[0042] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of a computing device, the processor in the computing device performs the above-described method.

[0043] This disclosure proposes a multimodal image-text interleaving generation model based on a dynamic feature synchronizer. The model includes an image encoder and a multimodal large language model. The image encoder extracts multi-resolution, multi-scale feature maps from the input image of the image-text interleaving data, which includes interleaved input images and input text. The dynamic feature synchronizer in the multimodal large language model obtains fine-grained information from the multi-resolution, multi-scale feature maps. This fine-grained information indicates the visual detail features of the multi-resolution, multi-scale feature maps, enabling the multimodal large language model to determine the output feature data corresponding to the image-text interleaving data based on the fine-grained information. The output feature data is used to generate target images and / or target text associated with the image-text interleaving data. In other words, by dynamically acquiring fine-grained information from the image-text interleaving data through the dynamic feature synchronizer in the multimodal large language model, the bottleneck of detail utilization is overcome, contextual understanding is improved, and efficient joint understanding and generation of image-text interleaving data are achieved, thereby comprehensively enhancing the overall performance of processing image-text interleaving data.

[0044] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0045] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0046] Figure 1 shows a schematic diagram of the structure of a computing device provided in an exemplary embodiment of the present disclosure;

[0047] Figure 2 shows a schematic diagram of the structure of a multimodal image-text interleaving generation model based on a dynamic feature synchronizer provided in an exemplary embodiment of this disclosure;

[0048] Figure 3 shows a flowchart of a multimodal image-text interleaving generation method based on a dynamic feature synchronizer provided in an exemplary embodiment of this disclosure;

[0049] Figure 4 shows a schematic diagram of the structure of a multimodal image-text interleaving generation model provided in another exemplary embodiment of this disclosure;

[0050] Figure 5 shows a schematic diagram of the principle of a dynamic feature synchronizer provided in an exemplary embodiment of the present disclosure;

[0051] Figure 6 shows a schematic diagram of the structure of a multimodal image-text interleaving generation apparatus based on a dynamic feature synchronizer provided in an exemplary embodiment of the present disclosure;

[0052] Figure 7 is a block diagram illustrating an apparatus for multimodal graphic interleaving generation according to an exemplary embodiment. Detailed Implementation

[0053] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0054] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0055] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0056] Current LLM-based solutions cannot effectively address the challenge of efficiently integrating detailed information from multiple high-resolution images within a multimodal large language model, especially in scenarios involving the generation of interleaved text-image sequences. Specifically, models like Flamingo and Language and Vision Alignment (LLaVA) connect LLM to a visual encoder, demonstrating strong text understanding and generation capabilities on paired data. However, they remain limited in image processing, failing to efficiently utilize image detail. Efficient Multimodal Understanding (Emu-2) models and Scalable Efficiently-fused Encoder-Decoder for Language and Math (SEED-LLaMA) primarily focus on understanding interleaved data, with generation capabilities limited to text. Some recent works, such as the CM3Leon and DreamLLM models, have made initial attempts to support interleaved text-image generation, but they still struggle with efficiently utilizing image detail.

[0057] In other words, the relevant technologies suffer from the following technical problems: 1. Loss of image details. Existing technologies compress and encode images into a fixed number of visual tokens for input into the model for processing. The limited number of tokens makes it difficult to retain all the detailed information of the original image. This deficiency becomes more severe in scenarios involving interleaved image and text data that require processing multiple high-resolution images. 2. Image features are not sensitive to context. Existing technologies only consider the information of the image itself when encoding images, ignoring the influence of contextual information. They cannot dynamically adjust the acquired visual features according to the context, making it difficult to efficiently utilize image details closely related to the current generation task. 3. Limited multimodal generation capabilities. Existing models mainly focus on tasks involving understanding interleaved image and text data. Their support for generation tasks, especially high-quality image generation, is relatively limited, and they generally lack the ability to jointly optimize image and text generation end-to-end.

[0058] This disclosure proposes a multimodal image-text interleaving generation framework based on a dynamic feature synchronizer. This framework overcomes the limitation of using fixed visual blocks to make it difficult to utilize image details. It can dynamically acquire fine-grained features of the input image in the intermediate layer of a multimodal large language model. That is, by dynamically acquiring fine-grained information related to the current task, it overcomes the bottleneck of detail utilization, realizes efficient joint understanding and generation of image-text interleaving data, and achieves the best performance in multiple benchmark tests. It improves contextual understanding ability and endows the model with the joint ability to optimize image and text generation end-to-end, thereby comprehensively enhancing the overall performance of processing image-text interleaving data.

[0059] First, the application scenarios involved in this disclosure will be introduced. Please refer to Figure 1, which shows a schematic diagram of the structure of a computing device provided in an exemplary embodiment of this disclosure.

[0060] Computing devices can be servers or terminals. Terminals include mobile terminals or fixed terminals, such as mobile phones, tablets, laptops, and desktop computers. A server can be a single server, a server cluster consisting of several servers, or a cloud computing service center.

[0061] The computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will understand that the structure shown in FIG1 does not constitute a limitation on the computing device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0062] The processor 10 is the control center of the computing device. It connects various parts of the computing device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 20, and by calling data stored in the memory 20, it performs various functions of the computing device and processes data, thereby controlling the computing device as a whole. The processor 10 can be implemented by a CPU or a GPU.

[0063] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. The program storage area may store the operating system 21, the acquisition module 22, the output module 23, the generation module 24, and at least one application program 25 required for a function; the data storage area may store data created according to the use of the computing device. The memory 20 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Accordingly, memory 20 may also include a memory controller to provide processor 10 with access to memory 20.

[0064] The processor 10 performs the following functions by running the acquisition module 22: acquiring image-text interleaved data, which includes interleaved input images and input text; the processor 10 performs the following functions by running the output module 23: based on the image-text interleaved data, using a trained multimodal large language model to output corresponding output feature data, where the dynamic feature synchronizer in the multimodal large language model is used to obtain fine-grained information from the multi-resolution, multi-scale feature maps of the input images, and the fine-grained information indicates the visual detail features of the multi-resolution, multi-scale feature maps; and the processor 10 performs the following functions by running the generation module 24: based on the output feature data, generating a target image and / or target text associated with the image-text interleaved data.

[0065] This disclosure proposes a multimodal image-text interleaving generation model based on a dynamic feature synchronizer. The core of this model lies in a dynamic feature synchronizer, which dynamically injects fine-grained information from high-resolution images into a multimodal large language model. This means that it can overcome the bottleneck of detail utilization, improve contextual understanding capabilities, and achieve efficient joint understanding and generation of image-text interleaving data, thereby comprehensively enhancing the overall performance in processing image-text interleaving data.

[0066] The multimodal text-image interleaving generation model proposed in this disclosure can be applied to multimodal text-image dialogue, text-image story generation, and other application scenarios. For example, in multimodal text-image dialogue applications, in chatbots or virtual assistants, the method proposed in this disclosure can be used to generate relevant images for interaction with users, enhancing the interactive experience. In text-image story generation applications, when automatically creating stories or comics, the model proposed in this disclosure can be used to ensure that the generated storyline closely matches the corresponding illustrations, improving the story's appeal and readability.

[0067] Please refer to Figure 2, which shows a schematic diagram of the structure of a multimodal image-text interleaving generation model based on a dynamic feature synchronizer provided in an exemplary embodiment of this disclosure.

[0068] The multimodal image-text interleaving generation model includes an image encoder 210 and a multimodal large language model 220. The multimodal large language model 220 includes a dynamic feature synchronizer 222. The image encoder 210 is used to extract multi-resolution, multi-scale feature maps from the input image of the image-text interleaving data, which includes interleaved input images and input text. The dynamic feature synchronizer 222 in the multimodal large language model 220 is used to obtain fine-grained information from the multi-resolution, multi-scale feature maps. The fine-grained information indicates the visual detail features of the multi-resolution, multi-scale feature maps, so that the multimodal large language model 220 determines the output feature data corresponding to the image-text interleaving data based on the fine-grained information. The output feature data is used to generate target images and / or target text associated with the image-text interleaving data.

[0069] The following describes the multimodal image-text interleaving generation method and multimodal image-text interleaving generation model based on a dynamic feature synchronizer provided in the embodiments of this disclosure using several exemplary embodiments.

[0070] Please refer to Figure 3, which shows a flowchart of a multimodal image-text interleaving generation method based on a dynamic feature synchronizer provided in an exemplary embodiment of this disclosure. The method includes the following steps.

[0071] Step 301: Obtain image-text interleaved data, which includes interleaved input images and input text.

[0072] Computing devices can acquire interlaced image and text data in various ways, including but not limited to the following: acquiring interlaced image and text data from a preset dataset; acquiring interlaced image and text data input by the user; acquiring interlaced image and text content uploaded by the user, collecting and processing the uploaded interlaced image and text content to obtain interlaced image and text data; or extracting interlaced image and text data from websites through web scraping technology, such as news articles, tutorials, etc., which usually naturally contain interlaced input images and input text.

[0073] Interleaved image data refers to data composed of interleaved input images and input text; that is, it is sequence data formed by interleaving input images and input text in any order. The input images can be at least one image or multiple images, and the input text can be at least one piece of text or multiple pieces of text.

[0074] In image-text interleaved data, there is a semantic relationship between the input text and the input images. When there are multiple input images and multiple segments of input text, there is a semantic relationship between at least one segment of input text and at least one input image in the image-text interleaved data. Optionally, the semantic relationship between the input text and the input image includes content matching, that is, the content described by the input text is consistent with the scene or object displayed by the input image. For example, if the input text describes an apple, then the corresponding input image is a picture of an apple.

[0075] Step 302: Based on the image-text interleaved data, the corresponding output feature data is obtained by using a multimodal large language model. The dynamic feature synchronizer in the multimodal large language model is used to obtain fine-grained information from the multi-resolution and multi-scale feature maps of the input image. The fine-grained information indicates the visual detail features of the multi-resolution and multi-scale feature maps.

[0076] Optionally, the computing device extracts visual feature vectors from the input image of the image-text interleaved data and extracts text feature vectors from the input text of the image-text interleaved data; based on the visual feature vectors and text feature vectors, it determines the input feature data corresponding to the image-text interleaved data; based on the input feature data and fine-grained information, it uses a multimodal large language model to output the corresponding output feature data.

[0077] A computing device can extract visual feature vectors from an input image of image-text interleaved data using an image encoder. In one possible implementation, the image encoder includes a pre-trained image feature encoder and a feature resampler. The computing device encodes the input image of the image-text interleaved data using the image feature encoder to obtain image semantic feature representations at different levels (shallow and deep); the feature resampler then compresses the image semantic features at the first resolution into a fixed number of visual feature vectors at a second resolution, which is lower than the first resolution.

[0078] The computing device can also extract multi-resolution, multi-scale feature maps from the input image of image-text interleaved data using an image encoder. These multi-resolution, multi-scale feature maps can be multiple image feature maps with different resolutions and levels. In one possible implementation, the image encoder also includes a multi-scale visual feature extractor. The computing device can use this extractor to extract fine-grained multi-resolution, multi-scale feature maps for use by the dynamic feature synchronizer in a subsequent multimodal large language model. Optionally, the computing device can use the multi-scale visual feature extractor to process the image semantic feature representation output by the image feature encoder based on a ViT adapter to extract fine-grained multi-resolution, multi-scale feature maps. The ViT adapter is a general-purpose adapter for processing visual tasks, used to extract multi-scale features through a module composed of a cross-attention mechanism and a feedforward neural network (FFN).

[0079] Computing devices can extract text feature vectors from input text in graph-text interleaved data through word embedding layers of multimodal large language models.

[0080] After obtaining the visual and text feature vectors, the computing device can concatenate them to obtain the input feature data corresponding to the image-text interleaved data. The concatenation method is consistent with the order of the input image and text in the image-text interleaved data. Then, the computing device can input the input feature data into the trained multimodal large language model and output the corresponding output feature data. The output feature data refers to the latent space state features output by the Transformer layer of the last layer of the multimodal large language model.

[0081] The multimodal large language model includes at least a dynamic feature synchronizer, a self-attention layer, and a feedforward network layer. The dynamic feature synchronizer dynamically injects extracted fine-grained information into the multimodal large language model. This fine-grained information indicates the visual detail features extracted from the multi-resolution, multi-scale feature maps of the input image. This fine-grained information participates in computation within the multimodal large language model; that is, the output feature data is determined jointly based on the input feature data and the fine-grained information. The fine-grained information extracted from the input image interacts with the textual semantic information in the context of the multimodal large language model, enabling it to more accurately capture richer local and detailed content in the input image, thus facilitating the generation of more accurate, coherent, semantically and stylistically consistent graphical and textual feedback.

[0082] Optionally, the dynamic feature synchronizer is located between the self-attention layer and the feedforward network layer in the multimodal large language model. The input parameters of the dynamic feature synchronizer are the hidden state features output by the self-attention module, and the output parameters, i.e., the fine-grained information, are state features with the same shape as the input parameters, obtained after fusing with the image features mentioned above. It should be noted that the relevant details of the dynamic feature synchronizer can be found in the relevant descriptions in the embodiments below, and will not be introduced here.

[0083] Step 303: Generate target images and / or target text associated with the image-text interleaving data based on the output feature data.

[0084] Computing devices can generate processing results associated with image-text interleaved data based on the output feature data of a multimodal large language model. These results can be target images and / or target text. The processing results can include the following forms: 1. Generating descriptive target text related to the content of the input image; this is often referred to as image annotation or image description. 2. Generating a target image that matches the input text description. 3. Generating a target image and / or target text that matches both the input image and input text. For example, a summary target text or a new target image can be generated based on the input image and input text. 4. In a multimodal question-answering system, the processing result is a target image and / or target text (i.e., the answer) generated based on the provided input image and input text (i.e., the question). For example, given an input image and a textual question, an answer in the form of an image or text related to the image-text information is generated. 5. The processing result is generated through an autoregressive approach. Autoregressive generation means that the output at each step of the generation process depends on the previously generated content. Indicatively, the processing result can be creative content. For example, given an input image and input text, the result can be used to create a story, poem, or image. In practical applications, these processing results can be used individually or combined to provide richer multimodal understanding and interaction capabilities.

[0085] In one possible implementation, a multimodal large language model processes both text and image data simultaneously. It uses a single model to predict the distribution of text tokens and samples the next text token—the target text—from this distribution. This process is autoregressive, meaning that the generation of each text token is based on a sequence of previously generated text. When the model generates a specific token, which is typically an indicator that the multimodal large language model should generate an image instead of continuing to generate text, the model switches to image generation mode and calls an image decoder to generate the next image. The image decoder can be another model specifically designed for image generation, generating a new target image based on the current text context and the content of already generated images.

[0086] Optionally, the computing device generates the target text corresponding to the image-text interleaved data through a text feature classifier based on the output feature data of the multimodal large language model.

[0087] Illustratively, a text feature classifier can be built using deep learning models or traditional machine learning methods. The task of this text feature classifier is to predict the target text based on the output feature data of a multimodal large language model. Existing image-text interleaving data and corresponding target text can be used as training data to train the text feature classifier. During the training process, the output feature data of the multimodal large language model can be used as input, and the target text as the label. During the inference process of the text feature classifier, for new image-text interleaving data, the multimodal large language model described above is first used to obtain the output feature data, and then the output feature data is input into the trained text feature classifier to obtain the predicted target text.

[0088] Optionally, the computing device generates the target image corresponding to the image-text interleaved data through an image decoder, based on the output feature data of the multimodal large language model and the multi-resolution, multi-scale feature maps extracted by the image encoder. The image decoder may include a dynamic feature synchronizer, which is used to fuse the features of the preceding image for image decoding.

[0089] This diagram illustrates the construction of an image decoder to generate the target image corresponding to image-text interleaved data based on the output feature data of a multimodal large language model and the multi-resolution, multi-scale feature maps extracted by an image encoder. This functionality can be achieved using techniques such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). Similarly, existing image-text interleaved data and corresponding target images can be used as training data to train the image decoder. During the training process, the output feature data and multi-resolution, multi-scale feature maps of the multimodal large language model can be used as input, and the target image can be used as the label. During the inference process of the image decoder, for new image-text interleaved data, the multi-resolution, multi-scale feature maps are first obtained using the image encoder described above, and the output feature data is obtained using the multimodal large language model described above. Then, the output feature data and the multi-resolution, multi-scale feature maps are input into the trained image decoder to obtain the predicted target image.

[0090] The multimodal image-text interleaving generation method provided in this disclosure can be entirely executed by a terminal device, entirely executed by a server, or partially executed by the terminal device and partially by the server. For example, the terminal device acquires input image-text interleaving data and sends the data to the server. The server executes the multimodal image-text interleaving generation method to generate a processing result corresponding to the image-text interleaving data. The processing result includes a target image and / or target text. The server then sends the processing result to the terminal device, which displays the result.

[0091] In summary, this disclosure proposes a multimodal image-text interleaving generation framework based on a dynamic feature synchronizer. It acquires image-text interleaving data, which includes interleaved input images and input text. Based on the image-text interleaving data, a multimodal large language model outputs corresponding output feature data. The dynamic feature synchronizer in the multimodal large language model is used to extract fine-grained information from the multi-resolution, multi-scale feature maps of the input images. This fine-grained information indicates the visual detail features of the multi-resolution, multi-scale feature maps. Based on the output feature data of the multimodal large language model, a target image and / or target text associated with the image-text interleaving data are generated. In other words, by dynamically acquiring fine-grained information of the input image through the dynamic feature synchronizer in the multimodal large language model, the bottleneck of detail utilization is overcome, and the contextual understanding capability is improved. This achieves efficient joint understanding and generation of image-text interleaving data, thereby comprehensively enhancing the overall performance of processing image-text interleaving data.

[0092] In one possible implementation, as shown in FIG4, the multimodal image-text interleaving generation model proposed in this disclosure includes, but is not limited to, an image encoder 210, a multimodal large language model 220, and an image decoder 230.

[0093] 1. The image encoder 210 may adopt the Visual Foundation Model (VFM) framework, that is, the image encoder 210 may be a VFM-based image encoder 210, which is used to extract visual representations from the input image of the image-text interleaved data.

[0094] In one possible implementation, the image encoder 210 includes a pre-trained image feature encoder, a feature resampling unit, and a multi-scale visual feature extractor.

[0095] (1) The image feature encoder can be implemented based on the Vision Transformer (ViT) network to encode the input image and obtain a series of image semantic feature representations.

[0096] (2) Feature resampling unit, used to compress high-dimensional image semantic features into a fixed number of low-resolution visual blocks. For example, the fixed number N is 32, and the number of visual image blocks is e. V .

[0097] (3) A multi-scale visual feature extractor is used to extract fine-grained multi-resolution, multi-scale feature maps for use by the dynamic feature synchronizer in the subsequent multimodal large language model 220. For example, the multi-resolution, multi-scale feature map is F V .

[0098] 2. The multimodal large language model 220 can adopt the LLM framework, that is, the multimodal large language model 220 can be an LLM-based multimodal large language model 220. The input parameters of the multimodal large language model 220 are input feature data composed of concatenated text feature vectors and visual feature vectors. The text feature vectors can be feature vectors extracted from the image-text interleaved data input text through the word embedding layer of the multimodal large language model 220, and the visual feature vectors can be feature vectors output by the image encoder 210. For example, the text feature vector is... Visual feature vector is The multimodal large language model 220 can adopt a standard Transformer architecture. The multimodal large language model 220 may include a multi-head self-attention layer, a dynamic feature synchronizer, a feedforward network layer, etc. The dynamic feature synchronizer in the multimodal large language model 220 is located between the self-attention layer and the feedforward network layer in the multimodal large language model 220.

[0099] 3. The image decoder 230 can adopt a diffusion model (DM) framework, meaning the image decoder 230 can be a DM-based image decoder 230. The image decoder 230 can be used to generate new images based on previous context. The image decoder 230 can sample images from noise based on interleaved graph data, which can contain any data modality, such as text, images, etc. In this framework, the input parameters of the image decoder 230 are: the output feature data output by the multimodal large language model 220, and the multi-resolution, multi-scale feature map output by the image encoder 210. The output parameter of the image decoder 230 is: the new image, i.e., the target image. The image decoder 230 may include an upsampling module, a dynamic feature synchronizer, and a downsampling module, etc. The dynamic feature synchronizer in the image decoder 230 is located between the upsampling module and the downsampling module in the image decoder 230.

[0100] The three modules described above (i.e., image encoder 210, multimodal large language model 220, and image decoder 230) achieve efficient modeling of alternating image and text sequences by fusing the visual representation generated by VFM with the text representation of LLM. The multimodal large language model 220 acts as a bridge for multimodal fusion, uniformly encoding inputs from both modalities into contextual features, thus providing the necessary conditions for both text generation and image generation tasks. The image decoder 230 is responsible for the final image generation, ensuring the semantic consistency of the generated image with the context.

[0101] In one possible implementation, the dynamic feature synchronizer provided in this disclosure embodiment can be a multi-scale multi-graph dynamic feature synchronizer (MMFS), which is based on deformable sparse attention. This allows the attention mechanism to move and sample feature values ​​on multi-resolution multi-scale feature maps, enabling multi-modal large language models and image decoders to dynamically obtain fine-grained information from the multi-resolution multi-scale feature maps extracted by the image encoder in the intermediate layer, avoiding information loss caused by fixed visual blocks in related technologies.

[0102] Optionally, the dynamic feature synchronizer is also used to extract fine-grained information from the multi-resolution multi-scale feature map based on the sampling point coordinates and corresponding attention weights of the multi-resolution multi-scale feature map. The fine-grained information includes the state features corresponding to each of the multiple sampling points sampled in the multi-resolution multi-scale feature map, and the state features can be feature vectors.

[0103] In one possible implementation, for each input image, the feature vector of each query token in the multi-resolution multi-scale feature map corresponding to the input image is determined; for each query token, the feature vector of the query token is linearly transformed and then combined with the image index embedding vector to obtain the target feature vector; based on the target feature vector, the coordinates of multiple sampling points and the corresponding attention weights in the multi-resolution multi-scale feature map are calculated through linear transformation and relative coordinate offset; the features of multiple sampling points with different coordinates are concatenated to obtain the overall sampling space point features, and the multiple attention weights are concatenated and normalized to obtain the target attention weights; based on the target attention weights and the sampling space point features, features are extracted and weighted from the multi-scale multi-resolution multi-scale feature map, and the output features obtained by the dynamic feature synchronizer are fused with the image features above to obtain the state features corresponding to each of the multiple sampling points in the multi-resolution multi-scale feature map of the input image.

[0104] Schematic, as shown in Figure 5, for each input image, based on the corresponding multi-resolution multi-scale feature map, the feature vector of each query token in the multi-resolution multi-scale feature map is determined. For each query token, the dynamic feature synchronizer only focuses on a small subset of sampling points near a reference point in the reference image. Let f q ∈R C R represents the feature vector of the query token. CThis refers to a real-valued domain space of dimension C. In multimodal large language models, a query token refers to any token in the context of the corresponding layer, which can be an image or text. The multi-resolution, multi-scale feature map is the object that the query token queries, not its source. This represents the relative coordinates of the reference point corresponding to the query token on a multi-resolution, multi-scale feature map. These coordinates are not absolute positions, but rather local offsets relative to the overall image. This method can be directly applied to scenarios with multiple feature maps because the relative coordinates are independent of the actual size of the feature maps. The advantage of this method is that it compresses the search space from a global location to a local relative offset, thereby reducing computation and improving efficiency. By default, if there is no spatial prior, it will... Set as the center of the multi-resolution, multi-scale feature map. It is a multi-resolution, multi-scale feature map extracted by the image encoder. M is the number of reference images, and the parameter images refer to all input images that appear before this input image in the image-text interleaving data. The output feature f of the dynamic feature synchronizer. o ∈R C The following formula is used to obtain: q (m) =W q ·f q +PosEmbed(m),

[0105] Among them, f q The latent space state features output by the self-attention module, i.e., the feature vector of the query token, q (m) This refers to the target feature vector obtained by linearly transforming the feature vector of the query token and adding the image index embedding vector. It is a learnable image index embedding vector, with a maximum of One reference image, W is an integer greater than or equal to 0. q W p W A All are learnable linear projective weights. The coordinates of sampling points in multi-resolution, multi-scale feature maps. For the corresponding attention weights, p is the coordinate of the reference point. q Concat is the concatenation operation used to obtain the features of the sampled spatial points. The coordinates of the first sampling point, Let A be the coordinates of the Mth sampling point, where m ranges from 1 to M, and M is a positive integer greater than 1. q These are the target attention weights obtained after concatenation and processing using the softmax function. For the first attention weight, For the Mth attention weight, the softmax function is a preset activation function; that is, multiple attention weights are normalized across multi-resolution, multi-scale feature maps using the softmax function. The DeformAttn operator is used to extract coordinates from the corresponding multi-resolution, multi-scale feature maps. The characteristics of the location, and according to A q ∈R M×L×K The output features of the dynamic feature synchronizer are obtained by weighted summation of the features corresponding to each coordinate in the coordinate sequence. L represents the number of multi-scale feature levels, and K represents the number of sampling points in the multi-resolution multi-scale feature map.

[0106] For multimodal large language models, each layer consists of multiple layers with the same structure, including a self-attention sublayer and a feedforward network layer. A dynamic feature synchronizer can be inserted between its self-attention and feedforward network layers at fixed intervals. The query token traverses each block in the multi-resolution, multi-scale feature map, but can only access previous images. To fuse the output features obtained from MMFS with the original features of the query token before querying MMFS, a residual connection method can be used. For example, the output features obtained from the dynamic feature synchronizer can be multiplied by tanh(α) and then added back to the original features, where α is a learnable scalar initialized to 0, and tanh is the hyperbolic tangent function.

[0107] For image decoders, a dynamic feature synchronizer can be inserted after each downsampling module in a U-Net of the diffusion model. The query token iterates through each block of the multi-resolution, multi-scale feature map. The spatial coordinates of the query token are set. The output features obtained from the dynamic feature synchronizer are processed by a convolution initialized to 0 and then added back to the original features. For the dynamic feature synchronizer in the image decoder, its input parameters (i.e., the query token) come from the features in the multi-resolution, multi-scale feature map output by the downsampling module; its output parameters are similar to those in the dynamic feature synchronizer in the large language model, and are also added back to the original features of the query token. The dynamic feature synchronizer in the image decoder is used to fuse the original features for image decoding; that is, the dynamic feature synchronizer in the image decoder is used to obtain fine-grained information from the multi-resolution, multi-scale feature map so that the image decoder can perform image decoding based on fine-grained information.

[0108] In summary, the embodiments of this disclosure construct an end-to-end multimodal image-text interleaving generation framework, integrating VFM, LLM, and DM to support end-to-end image-text generation training and preserve fine-grained information in images within a limited context. A core module, MMFS, is proposed, which effectively reduces the visual patch requirements of multimodal large language models and supports the dynamic extraction of fine-grained visual features from feature maps of different resolutions and scales across multiple images based on context. The framework and training method, trained on open-source image-text interleaving datasets and image-text pair datasets, demonstrate superior performance across numerous multimodal perception and generation benchmarks.

[0109] The following are device embodiments of the present disclosure. For parts not described in detail in the device embodiments, please refer to the technical details disclosed in the above method embodiments.

[0110] Please refer to Figure 6, which shows a schematic diagram of a multimodal image-text interleaving generation apparatus based on a dynamic feature synchronizer provided in an exemplary embodiment of this disclosure. This apparatus can be implemented as all or part of a computing device through software, hardware, or a combination of both. The apparatus includes: an acquisition module 22, an output module 23, and a generation module 24.

[0111] The acquisition module 22 is used to acquire image-text interleaved data, which includes interleaved input images and input text;

[0112] Output module 23 is used to output corresponding output feature data based on image-text interleaved data using a multimodal large language model. The dynamic feature synchronizer in the multimodal large language model is used to obtain fine-grained information from the multi-resolution and multi-scale feature maps of the input image. The fine-grained information indicates the visual detail features of the multi-resolution and multi-scale feature maps.

[0113] The generation module 24 is used to generate target images and / or target text associated with the image-text interleaving data based on the output feature data.

[0114] In one possible implementation, the dynamic feature synchronizer in the multimodal large language model is located between the self-attention layer and the feedforward network layer. The input parameters of the dynamic feature synchronizer include the latent space state features output by the self-attention module, and the output parameters of the dynamic feature synchronizer include state features with the same shape as the input parameters, which are obtained after being fused with the image features above.

[0115] In another possible implementation, the dynamic feature synchronizer in the multimodal large language model is also used to extract fine-grained information from the multi-resolution multi-scale feature map based on the coordinates of the sampling points and the corresponding attention weights in the multi-resolution multi-scale feature map. The fine-grained information includes the state features corresponding to each of the multiple sampling points.

[0116] In another possible implementation, output module 23 is also used for:

[0117] Visual feature vectors are extracted from the input images of the image-text interleaved data, and text feature vectors are extracted from the input text of the image-text interleaved data.

[0118] Based on the visual feature vector and the text feature vector, determine the input feature data corresponding to the image-text interleaved data;

[0119] Based on the input feature data, the corresponding output feature data is determined through a multimodal large language model.

[0120] In another possible implementation, generation module 24 is also used for:

[0121] Based on the output feature data and multi-resolution, multi-scale feature maps, the target image corresponding to the image-text interleaved data is generated by the image decoder.

[0122] In another possible implementation, the image decoder includes a dynamic feature synchronizer, which is used to obtain fine-grained information from multi-resolution, multi-scale feature maps so that the image decoder can perform image decoding based on the fine-grained information.

[0123] In another possible implementation, generation module 24 is also used for:

[0124] Based on the output feature data, the target text corresponding to the image-text interleaving data is generated through a text feature classifier.

[0125] It should be noted that the above embodiments only illustrate the division of the above functional modules when implementing the device. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0126] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0127] This disclosure also provides a multimodal image-text interleaving generation apparatus based on a dynamic feature synchronizer. The apparatus includes: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions stored in the memory.

[0128] This disclosure also provides a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the method described above.

[0129] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of a computing device, the processor in the computing device performs the above-described method.

[0130] Figure 7 is a block diagram illustrating an apparatus 1900 for multimodal graphics interleaving generation according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. Referring to Figure 7, apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by processing component 1922, such as application programs. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, processing component 1922 is configured to execute instructions to perform the methods described above.

[0131] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0132] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0133] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0134] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0135] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0136] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0137] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0138] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0139] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0141] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal image-text interleaving generation model based on a dynamic feature synchronizer, wherein, The multimodal image-text interleaving generation model includes an image encoder and a multimodal large language model; The image encoder is used to extract multi-resolution, multi-scale feature maps from the input image of the image-text interleaved data, which includes interleaved input images and input text. The dynamic feature synchronizer in the multimodal large language model is used to obtain fine-grained information from the multi-resolution multi-scale feature map. The fine-grained information indicates the visual detail features of the multi-resolution multi-scale feature map, so that the multimodal large language model determines the output feature data corresponding to the image-text interleaved data based on the fine-grained information. The output feature data is used to generate target images and / or target text associated with the image-text interleaved data.

2. The model according to claim 1, wherein, The dynamic feature synchronizer in the multimodal large language model is located between the self-attention layer and the feedforward network layer. The input parameters of the dynamic feature synchronizer include the latent space state features output by the self-attention module, and the output parameters of the dynamic feature synchronizer include state features with the same shape as the input parameters, which are obtained after being fused with the image features mentioned above.

3. The model according to claim 1 or 2, wherein, The dynamic feature synchronizer in the multimodal large language model is also used to extract the fine-grained information from the multi-resolution multi-scale feature map based on the sampling point coordinates and corresponding attention weights in the multi-resolution multi-scale feature map. The fine-grained information includes the state features corresponding to each of the multiple sampling points.

4. The model according to any one of claims 1 to 3, wherein, The image encoder is also used to extract visual feature vectors from the input image of the image-text interleaving data; The multimodal large language model is further used to extract text feature vectors from the input text of the image-text interleaved data; determine the input feature data corresponding to the image-text interleaved data based on the visual feature vectors and the text feature vectors; and determine the corresponding output feature data based on the input feature data and the fine-grained information.

5. The model according to any one of claims 1 to 3, wherein, The multimodal image-text interleaving generation model further includes: an image decoder; The image decoder is used to generate the target image corresponding to the image-text interleaving data based on the output feature data and the multi-resolution multi-scale feature map.

6. The method according to claim 5, wherein, The image decoder includes a dynamic feature synchronizer, which is used to obtain the fine-grained information from the multi-resolution, multi-scale feature map so that the image decoder can perform image decoding based on the fine-grained information.

7. The model according to any one of claims 1 to 3, wherein, The multimodal image-text interleaving generation model also includes: a text feature classifier; The text feature classifier is used to generate the target text corresponding to the image-text interleaving data based on the output feature data.

8. A multimodal image-text interleaving generation method based on a dynamic feature synchronizer, wherein, The method includes: Acquire image-text interleaved data, which includes interleaved input images and input text; Based on the image-text interleaved data, the corresponding output feature data is obtained by using a multimodal large language model. The dynamic feature synchronizer in the multimodal large language model is used to obtain fine-grained information from the multi-resolution multi-scale feature map of the input image. The fine-grained information indicates the visual detail features of the multi-resolution multi-scale feature map. Based on the output feature data, a target image and / or target text associated with the image-text interleaving data are generated.

9. A multimodal image-text interleaving generation device based on a dynamic feature synchronizer, wherein, The device includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of claim 8 when executing instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein, When the computer program instructions are executed by the processor, they implement the method of claim 8.

Images