Visual narrative generation methods, devices, electronic devices, and storage media based on visual language alignment

Abstract

This application discloses a visual narrative generation method, apparatus, electronic device, and storage medium based on visual-language alignment, belonging to the field of image understanding technology. The method includes: acquiring a sequence of images to be processed and instruction text; inputting the sequence of images to be processed and the instruction text into a multimodal large model to obtain visual narrative text output by the multimodal large model; the multimodal large model includes a visual encoder, a multimodal mapper, and a large language model connected in sequence, and also includes a text encoder, the output of which is connected to the input of the large language model; the step of inputting the sequence of images to be processed and the instruction text into the multimodal large model to obtain the visual narrative text output by the multimodal large model includes: sequentially encoding the images in the sequence of images to be processed using the visual encoder to generate visual features; sequentially projecting the visual features into the word embedding space of the large language model using the multimodal mapper for semantic alignment to obtain language embedding vectors; converting the instruction text into instruction embedding vectors using the text encoder; and generating the visual narrative text by performing logical reasoning based on the language embedding vectors and the instruction embedding vectors using the large language model. This method can effectively align visual information with language output, improving the visual relevance of multi-image narratives.

Description

Technical Field

[0001] This application belongs to the field of image understanding technology, and in particular relates to a visual narrative generation method, device, electronic device and storage medium based on visual language alignment. Background Technology

[0002] Humans can easily identify objects and targets in a given image, understand the logical relationships between them, and describe the content to someone who has never seen the image before, even supplementing it with common sense and external knowledge. Enabling computers to possess similar capabilities is a very challenging task.

[0003] Visual storytelling is a task that uses non-textual data such as images as input to generate textual stories related to information in the images or visual concepts. With the development of multimodal large language models, visual storytelling has become an important direction in content creation. Multimodal text generation not only requires a systematic understanding of image content, but also syntactic and grammatical parsing of the text, and effective fusion of the two modalities to achieve multimodal reasoning, making the text generation task highly challenging.

[0004] Currently, visual generative models based on the Transformer architecture and large language and vision assistant (LLaVA) can be applied to generate narrative content by combining vision and language. However, when generating image sequence narratives, they often contain details that are not related to the images, resulting in a lack of visual relevance in the generated content, difficulty in maintaining the coherence of the plot, and inconsistency between the narrative content and the timeline of the image sequence. Therefore, the generated narratives are difficult to achieve the richness and appeal of human creation, and the stories lack emotional resonance and in-depth plot portrayal.

[0005] Therefore, there is an urgent need for a new method for generating visual narratives that can effectively align visual information with linguistic output, thereby improving the narrative quality and consistency of visual story creation. Summary of the Invention

[0006] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a visual narrative generation method, apparatus, electronic device, and storage medium based on visual-language alignment, which can effectively align visual information with language output and improve the visual relevance of multi-image narratives.

[0007] Firstly, this application provides a visual narrative generation method based on visual language alignment, the method comprising:

[0008] Obtain the image sequence to be processed and the instruction text;

[0009] The image sequence to be processed and the instruction text are input into the multimodal large model to obtain the visual narrative text output by the multimodal large model;

[0010] The multimodal large model includes a visual encoder, a multimodal mapper, and a large language model connected in sequence, and also includes a text encoder, the output of which is connected to the input of the large language model;

[0011] The step of inputting the image sequence to be processed and the instruction text into a multimodal large model to obtain the visual narrative text output by the multimodal large model includes:

[0012] The visual encoder sequentially encodes the images in the image sequence to be processed to generate visual features.

[0013] The visual features are sequentially projected into the word embedding space of the large language model through the multimodal mapper for semantic alignment, resulting in language embedding vectors.

[0014] The text encoder converts the instruction text into an instruction embedding vector.

[0015] The visual narrative text is generated by performing logical reasoning based on the language embedding vector and the instruction embedding vector using the large language model.

[0016] According to one embodiment of this application, before inputting the image sequence to be processed and the instruction text into the multimodal large model, the method further includes:

[0017] Obtain multiple image sequence samples from the sample set, and an initial storyline for the multiple image sequence samples, wherein the image sequence samples include multiple sample images;

[0018] Generate an overall description for each of the sample images in the image sequence samples;

[0019] Each of the sample images is divided into local regions and a corresponding local description is generated for each.

[0020] Based on the overall description and the local description, an image description of the sample image is obtained;

[0021] Based on the initial storyline of the image sequence samples and the image description of each sample image in the image sequence samples, the image theme of the image sequence samples is determined;

[0022] The initial storyline, the image description, and the image theme are merged to generate story narrative tags for the image sequence samples;

[0023] The combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text is used as training samples and input into the multimodal large model to train the multimodal large model.

[0024] According to one embodiment of this application, determining the image theme of the image sequence sample based on the initial storyline of the image sequence sample and the image description of each sample image in the image sequence sample includes:

[0025] Based on the image description and the initial storyline, a first theme is generated for each image sequence sample;

[0026] Based on the data source of the image sequence samples, the first topic is divided into at least one topic set;

[0027] The first topic in the topic set is deduplicated and filtered to obtain at least one second topic;

[0028] Based on the initial storyline, the image theme of each image sequence sample is determined by matching the second theme with the corresponding image sequence samples from the data source.

[0029] According to one embodiment of this application, based on the initial storyline, the image theme of each image sequence sample is determined by performing theme matching between a second theme and corresponding image sequence samples from the data source, including:

[0030] The plurality of image sequence samples are returned to the sample set;

[0031] The image sequences in the sample set are divided according to their data sources;

[0032] By matching the initial storyline of the image sequence samples with the second theme corresponding to the data source, the image theme of the image sequence samples is determined.

[0033] The image subject is generated based on the initial storyline and the image description of the unmatched image sequence samples.

[0034] According to one embodiment of this application, before inputting the combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text as training samples into a multimodal large model to train the multimodal large model, the method further includes:

[0035] Acquire multiple image-text pairs, wherein each image-text pair includes an image file and a corresponding image description text;

[0036] The image-text pair is input into the multimodal large model to train the modality mapper in the multimodal large model.

[0037] According to one embodiment of this application, the combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text is used as training samples and input into a multimodal large model to train the multimodal large model, including:

[0038] Using the training samples, the probability of the multimodal large model generating story narratives is determined:

[0039]

[0040] Among them, s k It is the output of the multimodal large model based on the k-th training sample; in the sample instruction text, X Instrution This refers to the sample instruction text specifying the output content; K is the generated s k Style type; r i It is the generated s k Word count requirement; The input is an image sequence sample, which includes m sample images, X′ vj It is the j-th sample image in the image sequence sample;

[0041] The parameters of the large language model in the multimodal large model are fine-tuned based on the probability.

[0042] According to one embodiment of this application, after inputting the combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text as training samples into a multimodal large model to train the multimodal large model, the method further includes:

[0043] The training samples are cyclically input into the multimodal large model to obtain multiple narrative outputs from the multimodal large model;

[0044] The sample images in the training samples are reordered and input into the multimodal large model to obtain multiple narrative outputs from the multimodal large model;

[0045] The multiple narrative outputs are scored and ranked, and the highest-ranked narrative output is used as a positive example of the image sequence samples in the training samples, while the lowest-ranked narrative output is used as a negative example of the image sequence samples in the training samples.

[0046] The multimodal large model is optimized by using the training samples to correspond to the positive and negative examples.

[0047] Secondly, this application provides a visual narrative generation device based on visual language alignment, the device comprising:

[0048] The acquisition module is used to acquire the image sequence to be processed and the instruction text;

[0049] The processing module is used to input the image sequence to be processed and the instruction text into the multimodal large model to obtain the visual narrative text output by the multimodal large model;

[0050] The multimodal large model includes a visual encoder, a multimodal mapper, and a large language model connected in sequence, and also includes a text encoder, the output of which is connected to the input of the large language model;

[0051] The processing module is further used for:

[0052] The visual encoder sequentially encodes the images in the image sequence to be processed to generate visual features.

[0053] The visual features are sequentially projected into the word embedding space of the large language model through the multimodal mapper for semantic alignment, resulting in language embedding vectors.

[0054] The text encoder converts the instruction text into an instruction embedding vector.

[0055] The visual narrative text is generated by performing logical reasoning based on the language embedding vector and the instruction embedding vector using the large language model.

[0056] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual narrative generation method based on visual language alignment as described in the first aspect above.

[0057] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the visual narrative generation method based on visual language alignment as described in the first aspect above.

[0058] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the visual narrative generation method based on visual language alignment as described in the first aspect.

[0059] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the visual narrative generation method based on visual language alignment as described in the first aspect above.

[0060] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0061] This application provides a visual narrative generation method, apparatus, electronic device, and storage medium based on visual language alignment, which have the following advantages over the prior art:

[0062] (1) Through the modal mapper, visual features can be projected sequentially into the word embedding space of the large language model, which can adapt to the visual narrative generation task, effectively align visual information with language output, improve the narrative quality and consistency of visual story creation, and the visual narrative text output by the large language model has visual correlation with the input image sequence and instruction text, which can maintain the coherence of the plot, and the narrative content is consistent with the time axis of the image sequence, thus achieving high-quality visual narrative. It can be widely applied in the fields of intelligent content creation, virtual narrative generation and multimodal artificial intelligence.

[0063] (2) Through the multimodal large model, rich story data optimization strategies such as image description generation, theme generation and story generation are used to extract deep visual semantics and logical story evolution, ensuring that the theme of the narrative content is clear and logically coherent. Through three-stage training of visual language alignment pre-training, story data fine-tuning and human preference optimization, the coherence, visual relevance and human preference adaptability of the visual narrative text generated by the multimodal large model are significantly improved, and the generation of high-quality text narrative from visual information is completed. The generated visual narrative has been significantly improved in terms of theme coherence, detail richness and human aesthetic adaptability, generating more attractive story content that conforms to human preferences, and improving the quality of visual narrative generation. Attached Figure Description

[0064] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0065] Figure 1 This is one of the flowcharts illustrating the visual narrative generation method based on visual language alignment provided in the embodiments of this application;

[0066] Figure 2 This is the second flowchart of the visual narrative generation method based on visual language alignment provided in the embodiments of this application;

[0067] Figure 3 It is a schematic diagram of the visual narrative provided by the related technology;

[0068] Figure 4 This is a model framework diagram of the multimodal large model provided in the embodiments of this application;

[0069] Figure 5This is a flowchart of the story data enrichment method provided in the embodiments of this application;

[0070] Figure 6 This is a schematic diagram of the three-stage training process provided in the embodiments of this application;

[0071] Figure 7 This is a schematic diagram of the generation result of the multimodal large model provided in the embodiments of this application;

[0072] Figure 8 This is a schematic diagram of the structure of the visual narrative generation device based on visual language alignment provided in the embodiments of this application;

[0073] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0074] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0075] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0076] The following description, in conjunction with the accompanying drawings, details the visual narrative generation method, visual narrative generation device, electronic device, and readable storage medium based on visual language alignment provided in this application, through specific embodiments and application scenarios.

[0077] Among them, the visual narrative generation method based on visual language alignment can be applied to the terminal, specifically executed by the hardware or software in the terminal.

[0078] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0079] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0080] The visual narrative generation method based on visual language alignment provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the visual narrative generation method based on visual language alignment. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The following uses an electronic device as the execution subject to describe the visual narrative generation method based on visual language alignment provided in this application embodiment.

[0081] like Figure 1 As shown, this visual narrative generation method based on visual language alignment includes:

[0082] Step 110: Obtain the image sequence to be processed and the instruction text;

[0083] Step 120: Input the image sequence to be processed and the instruction text into the multimodal large model to obtain the visual narrative text output by the multimodal large model;

[0084] The multimodal large model includes a visual encoder, a multimodal mapper, and a large language model connected in sequence, and also includes a text encoder, the output of which is connected to the input of the large language model;

[0085] like Figure 2 As shown, step 120, which involves inputting the image sequence to be processed and the instruction text into a multimodal large model to obtain the visual narrative text output by the multimodal large model, includes:

[0086] Step 121a: The images in the image sequence to be processed are encoded sequentially by the visual encoder to generate visual features;

[0087] Step 122: Using the multimodal mapper, the visual features are sequentially projected into the word embedding space of the large language model for semantic alignment to obtain language embedding vectors;

[0088] Step 121b: Convert the instruction text into an instruction embedding vector using the text encoder;

[0089] Step 123: Using the large language model, perform logical reasoning based on the language embedding vector and the instruction embedding vector to generate the visual narrative text.

[0090] like Figure 3 As shown, given an image, humans can easily identify the objects and targets within it, understand the logical relationships between them, and describe its content to someone who has never seen the image before. They can even supplement this with common sense and external knowledge, achieving visual embedding. In the embodiments of this application, a similar effect is achieved by setting up a multimodal large model.

[0091] like Figure 4 As shown, the multimodal large model can use the pre-trained SigLIP-so400m as a visual encoder to process the input image sequence at a high resolution of 384×384, generate feature maps from each image as visual features, and convert visual information into a representation compatible with the language model.

[0092] By using two layers of multilayer perceptrons (MLPs) as multimodal projectors, visual features are projected onto the word embedding space of the language model for semantic alignment, generating language embedding vectors with the same spatial dimension as the word embedding space, enabling the fusion of visual information with textual information.

[0093] The large language model uses the Phi-3 Mini-128K model, which can support up to 128K tokens to ensure effective processing of multiple image inputs and reduce computational resource consumption.

[0094] In actual execution, the sequence of images to be processed can be a set of n consecutively input images. Let i be the i-th image, i∈n. Each image is encoded using a visual encoder to obtain its corresponding visual features. It is the visual feature of the i-th image.

[0095] These visual features are used as a multimodal mapper through a trainable two-layer MLP. Convert to language embedding vectors with dimensions consistent with the word embedding space of large language models:

[0096]

[0097] in, It is the language embedding vector of the i-th image.

[0098] The user's language instruction is vectorized using a text encoder. The instruction text is then preprocessed using tokenizer and embedding techniques to obtain information such as the instruction, topic indices, and story requirements. This information is then vectorized to obtain the instruction embedding vector H. c .

[0099] Visual storytelling is achieved through a large language model, based on language embedding vectors. And let the embedding vector H c To infer the visual narrative text X corresponding to the image S , can be represented as:

[0100]

[0101] In large multimodal models, multimodal mappers can adapt to visual narrative generation tasks by projecting visual features into the word embedding space of large language models, ensuring the efficient fusion of visual and linguistic information.

[0102] According to the visual narrative generation method based on visual-language alignment provided in the embodiments of this application, a modal mapper can sequentially project visual features into the word embedding space of a large language model. This method can adapt to visual narrative generation tasks, effectively align visual information with language output, improve the narrative quality and consistency of visual story creation, and ensure that the visual narrative text output by the large language model has visual relevance to the input image sequence and instruction text, thus maintaining the coherence of the plot. The narrative content is consistent with the timeline of the image sequence, achieving high-quality visual narrative. This method can be widely applied in the fields of intelligent content creation, virtual narrative generation, and multimodal artificial intelligence.

[0103] In some embodiments, before inputting the image sequence to be processed and the instruction text into the multimodal large model, the method further includes:

[0104] Obtain multiple image sequence samples from the sample set, and an initial storyline for the multiple image sequence samples, wherein the image sequence samples include multiple sample images;

[0105] Generate an overall description for each of the sample images in the image sequence samples;

[0106] Each of the sample images is divided into local regions and a corresponding local description is generated for each.

[0107] Based on the overall description and the local description, an image description of the sample image is obtained;

[0108] Based on the initial storyline of the image sequence samples and the image description of each sample image in the image sequence samples, the image theme of the image sequence samples is determined;

[0109] The initial storyline, the image description, and the image theme are merged to generate story narrative tags for the image sequence samples;

[0110] The combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text is used as training samples and input into the multimodal large model to train the multimodal large model.

[0111] The sample instruction text can be determined based on the relationship between the image sequence samples and the story narrative labels.

[0112] In some embodiments, obtaining multiple image sequence samples from a sample set includes:

[0113] Multiple image sequences were obtained from the VIST, VWP, PororoSV, and FlintstonesSV datasets as image sequence samples to construct a sample set.

[0114] Based on the data source, the multiple image sequence samples are extracted from the sample set.

[0115] Based on the size of the original dataset, image sequence samples were selected from different datasets. For example, 20k, 10k, 5k and 5k image sequence samples were selected from datasets such as VIST, VWP, PororoSV and FlintstonesSV, respectively, for low-rank adaptive (Lora) fine-tuning and data optimization, ultimately generating a total of 40k image sequence samples for fine-tuning.

[0116] During the narrative generation process, image themes are automatically extracted from image sequence samples and optimized using the GPT-4 model to ensure that the generated narrative content has a clear theme and avoids redundant and irrelevant details. By generating multiple versions of the story and ranking these versions using human preference data, the narrative that best conforms to human aesthetics is selected to optimize the quality of the model-generated content.

[0117] In actual execution, each image sequence sample is processed to obtain the initial storyline corresponding to the image sequence sample. For example, an image sequence sample includes 5 sample images.

[0118] like Figure 5As shown, based on caption generation technology and combined with local captions, rich and high-quality caption data can significantly improve the capabilities of multimodal models.

[0119] In the overall caption stage, a detailed caption is generated for each sample image in the image sequence using a visual base model such as Florence-2. At the same time, a segmentation model (SAM) is used to divide each sample image into local regions, generating a detailed local description for each region in the sample image.

[0120] To make these descriptions more closely resemble human language, a phrase-level generation strategy can be used to refine the overall and local descriptions of each sample image, ultimately resulting in a detailed and structured image description for each sample image, providing comprehensive information support for the next step of topic generation.

[0121] In some embodiments, determining the image theme of the image sequence sample based on the initial storyline of the image sequence sample and the image description of each sample image in the image sequence sample includes:

[0122] Based on the image description and the initial storyline, a first theme is generated for each image sequence sample;

[0123] Based on the data source of the image sequence samples, the first topic is divided into at least one topic set;

[0124] The first topic in the topic set is deduplicated and filtered to obtain at least one second topic;

[0125] Based on the initial storyline, the image theme of each image sequence sample is determined by matching the second theme with the corresponding image sequence samples from the data source.

[0126] In the process of image topic generation (Story Topic Generation), the TopicGPT model is used to generate the first topic related to the initial storyline based on the image description. Since different image sequence samples are obtained from different datasets and have different data sources, the first topic can be divided into different topic sets according to different data sources. Each topic set corresponds to a data source. The first topic in the topic set is processed by deduplication, merging duplicate topics, and removing rare topics to obtain the second topic corresponding to each data source, ensuring quality and coverage, and making the topics more accurate and diverse.

[0127] In some embodiments, based on the initial storyline, the image theme of each image sequence sample is determined by performing theme matching between a second theme and corresponding image sequence samples from the data source, including:

[0128] The plurality of image sequence samples are returned to the sample set;

[0129] The image sequences in the sample set are divided according to their data sources;

[0130] By matching the initial storyline of the image sequence samples with the second theme corresponding to the data source, the image theme of the image sequence samples is determined.

[0131] The image topic is generated based on the initial storyline and image description of the unmatched image sequence samples.

[0132] The extracted image sequence samples are put back into the sample set. The image sequence samples in the sample set are divided according to the data source. According to the assignment prompt, the image sequence samples of the corresponding data source are matched with the second topic according to the initial storyline. Some image sequence samples can be successfully matched with the second topic, and the second topic is used as the image topic of the image sequence sample.

[0133] For image sequence samples without a matching topic, a predefined generation prompt from the TopicGPT model is used to generate a new image topic based on the initial storyline and the image description, ensuring that the initial storyline of each image sequence sample has a clear direction. When computational costs need to be reduced, open-source models such as Mistral-7B can be used instead of GPT-4 for topic assignment, thereby achieving localized computation.

[0134] The initial storyline, image description, and image theme of each image sequence sample are fused. The GPT model is used to focus on the logical coherence and thematic consistency of the narrative, generating a high-quality, complete narrative that combines visual information and theme, avoiding false details irrelevant to the image content or thematic direction. By integrating visual details and theme guidance, the generated narrative not only accurately reflects the information of the input image but is also more emotionally and logically appealing, outputting a high-quality story S.

[0135] S = GPT(y, C, T, Prompt)

[0136] Where y is the initial storyline, C is the image description, T is the image subject, and the prompt guides GPT-4 to integrate these inputs into a coherent story S.

[0137] The story S is used as the story narrative label of the corresponding image sequence sample. The combination of the image sequence sample and the corresponding story narrative label is used as a training sample. Multiple training samples can be obtained. Multiple training samples are then input into the multimodal large model for training.

[0138] In this embodiment, by enriching story data, stories can be automatically rewritten and enriched, integrating deeper visual semantics and logical story evolution; by generating image descriptions, detailed visual information support is provided for subsequent topic generation and story generation; content-related topic sets are extracted or generated from image descriptions to provide directional support for story narration; and logical and attractive story narration tags are generated based on image descriptions, topic sets, and initial storylines, providing a foundation for the training of multimodal large models.

[0139] like Figure 6 As shown, high-quality story generation is achieved through multi-stage training and inference. In the multi-stage training, the multimodal large model is gradually optimized in each stage through three stages: image and text pre-training, story data fine-tuning training, and preference optimization training, so as to improve the quality of visual narrative generation and ultimately generate high-quality visual narrative content.

[0140] In some embodiments, before training the multimodal large model, the method further includes: inputting the image sequence samples, the corresponding story narrative tags, and the sample instruction text as training samples into the multimodal large model.

[0141] Acquire multiple image-text pairs, wherein each image-text pair includes an image file and a corresponding image description text;

[0142] The image-text pair is input into the multimodal large model to train the modality mapper in the multimodal large model.

[0143] In this embodiment, through pre-training of the multimodal large model, the modality mapper learns to align visual features to the word embedding space of the large language model, and the large language model (LLM) is able to understand the output of the modality mapper.

[0144] In some embodiments, the combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text is used as training samples and input into a multimodal large model to train the multimodal large model, including:

[0145] Using the training samples, the probability of the multimodal large model generating story narratives is determined:

[0146]

[0147] Among them, s k It is the narrative output of the multimodal large model based on the k-th training sample; in the sample instruction text, X Insrrution It is the sample instruction text for the output content, K is the generated s k Style type; r i It is the generated s k Word count requirement; The input is an image sequence sample, which includes m sample images, X′ vj It is the j-th sample image in the image sequence sample;

[0148] The parameters of the large language model in the multimodal large model are fine-tuned based on the probability.

[0149] Multimodal large models learn reinforcement from human feedback, effectively guiding large language models to generate more honest, useful, and harmless content. This enables multimodal large models not only to learn to match image sequences but also to learn to adjust the storyline appropriately in different image arrangements, thereby improving the overall fluency of the story.

[0150] In this embodiment, by supervising the fine-tuning of the parameters of the large language model, the probability of generating story narratives is maximized, enabling the multimodal large model to enhance its understanding of images and generate more coherent stories based on visual-language alignment.

[0151] Through fine-tuning, the length of the story generated by the multimodal large model was increased, but the hallucination of the multimodal large model increased significantly, requiring further optimization of the multimodal large model.

[0152] In some embodiments, after inputting the combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text as training samples into a multimodal large model to train the multimodal large model, the method further includes:

[0153] The training samples are cyclically input into the multimodal large model to obtain multiple narrative outputs from the multimodal large model;

[0154] The sample images in the training samples are reordered and input into the multimodal large model to obtain multiple narrative outputs from the multimodal large model;

[0155] The multiple narrative outputs are scored and ranked, and the highest-ranked narrative output is used as a positive example of the image sequence samples in the training samples, while the lowest-ranked narrative output is used as a negative example of the image sequence samples in the training samples.

[0156] The multimodal large model is optimized by using the training samples to correspond to the positive and negative examples.

[0157] After training, the six narrative outputs generated by the multimodal large model for each training sample are evaluated. Then, the order of the sample images in each training sample is swapped, and the sampling continues to generate narrative outputs. All narrative outputs are scored and ranked according to preset quality standards. The narrative output with the best score and the highest ranking is selected as a positive example, and the narrative output with the lowest score and the lowest ranking is selected as a negative example.

[0158] Combinations of image sequences, chosen stories, and rejected stories from the training samples were used as preference data, resulting in a total of 6,000 preference data entries. Each data entry contains...<imagesequence,chosen story,rejected story> The samples, along with the corresponding sample instruction text, are used for Direct Preference Optimization (DPO) training to optimize large multimodal models.

[0159] In some embodiments, the training objective is defined as:

[0160]

[0161] Where, π θ It is a multimodal large model in DPO training; π ref This is the fine-tuned baseline reference model, where I is the image sequence and S is... w It's the chosen story, S l It's a story of rejection, E(I,S) w ,S l ) represents the expected value of the image sequence and story pair on the dataset, σ is a logistic function, and β is a parameter set to 0.1.

[0162] In this embodiment, preference data is used to optimize the model through the DPO method, making the stories generated by the multimodal large model more in line with human preferences. After adjusting the model parameters of the multimodal large model, the generated stories can achieve optimal coherence, non-redundancy and fluency, achieving a high-quality narrative effect. It is significantly better than the existing technology in terms of visual narrative coherence, fluency and non-redundancy.

[0163] The existing structurally similar models and the multimodal large model provided in the embodiments of this application were validated on the VIST dataset. The experimental results are as follows: Figure 7 As shown, the results of the multimodal large model of this application generated by sampling test set images on a cartoon dataset are qualitatively presented. The experimental results show that the multimodal large model of this application achieves the best performance in narrative text generation.

[0164] Table 1 shows the no-reference metrics on the VIST dataset.

[0165] AREL 23.5 0.584 0.577 0.833 MCSM+BART 2.8 0.852 0.666 0.865 TAPM 6.8 0.734 0.671 0.903 LLaVA 6.5 0.541 0.809 0.851 Multimodal large model 0.5 0.764 0.833 0.905

[0166] In the absence of a reference metric, the lower the Intra-Repetition value, the higher the quality of the story.

[0167] GROOViST (Visual Relevance Score) is a metric used to assess the alignment between visuals and text in visual storytelling. It calculates a visual alignment score by extracting noun phrases from the story and comparing them to visual objects in images. Compared to existing metrics, it aligns more closely with human intuition about visual associations, making it a highly interpretable and scalable assessment tool. A higher GROOViST score indicates a higher quality story.

[0168] RoViST-C is used to evaluate the inter-sentence coherence of a story. The model captures the logical flow of a story by judging whether adjacent sentences are coherent, and calculates the coherence probability of each pair of adjacent sentences, ultimately generating an overall coherence score. It effectively captures natural transitions between sentences and is suitable for analyzing the structural integrity of generated stories. The higher the RoViST-C coherence, the higher the story quality.

[0169] RoViST-NR is used to assess the redundancy of a story, including inter-sentence and intra-sentence repetition. The degree of repetition is measured by calculating the Jaccard similarity of words between and within sentences. A higher score indicates less redundancy and higher story quality.

[0170] As shown in Table 1, in the results of applying the no-reference automatic evaluation metric on the VIST test set, which uses five consecutive images without real label annotations, the multimodal large model of this application consistently outperforms the AREL model, MCSM+BART model, TAPM model and LlaVA model in terms of coherence, non-redundancy and fluency.

[0171] Table 2 provides reference metrics for the VIST dataset.

[0172] AREL 35.2 29.3 9.1 MCSM+BART 36.1 30.7 11.0 TAPM 33.1 37.2 13.8 Multimodal large model 29.9 33.7 14.5

[0173] Among the available metrics, the higher the METEOR (Metric for Evaluation of Translation with Explicit Ordering), the higher the quality of the story; the higher the ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation - Longest common subsequence), the higher the quality of the story; and the higher the CIDEr (Cosensus-based Image Description Evaluation), the higher the quality of the story.

[0174] As shown in Table 2, in the results of applying the reference automatic evaluation metric on the VIST test set, the multimodal large model of this application performs well on the traditional word overlap-based evaluation metric.

[0175] In this embodiment, a multimodal large model is used to optimize rich story data such as image description generation, theme generation, and story generation. Deep visual semantics and logical story evolution are extracted to ensure that the narrative content has a clear theme and logical coherence. Through three-stage training, namely visual language alignment pre-training, story data fine-tuning, and human preference optimization, the coherence, visual relevance, and human preference adaptability of the visual narrative text generated by the multimodal large model are significantly improved. This completes the generation of high-quality text narrative from visual information, and significantly improves the generated visual narrative in terms of theme coherence, detail richness, and human aesthetic adaptability. The resulting story content is more engaging and conforms to human preferences, thus improving the quality of visual narrative generation.

[0176] The visual narrative generation method based on visual language alignment provided in this application can be executed by a visual narrative generation device based on visual language alignment. This application uses the execution of the visual narrative generation method based on visual language alignment by a visual narrative generation device as an example to illustrate the visual narrative generation device based on visual language alignment provided in this application.

[0177] This application also provides a visual narrative generation device based on visual language alignment.

[0178] like Figure 8 As shown, the visual narrative generation device based on visual language alignment includes:

[0179] The acquisition module 810 is used to acquire the image sequence to be processed and the instruction text;

[0180] The processing module 820 is used to input the image sequence to be processed and the instruction text into the multimodal large model to obtain the visual narrative text output by the multimodal large model;

[0181] The multimodal large model includes a visual encoder, a multimodal mapper, and a large language model connected in sequence, and also includes a text encoder, the output of which is connected to the input of the large language model;

[0182] The processing module 820 is further configured to:

[0183] The visual encoder sequentially encodes the images in the image sequence to be processed to generate visual features.

[0184] The visual features are sequentially projected into the word embedding space of the large language model through the multimodal mapper for semantic alignment, resulting in language embedding vectors.

[0185] The text encoder converts the instruction text into an instruction embedding vector.

[0186] The visual narrative text is generated by performing logical reasoning based on the language embedding vector and the instruction embedding vector using the large language model.

[0187] According to the visual narrative generation device based on visual language alignment provided in the embodiments of this application, a modal mapper can sequentially project visual features into the word embedding space of a large language model. This adapts to visual narrative generation tasks, effectively aligns visual information with language output, and improves the narrative quality and consistency of visual story creation. The visual narrative text output by the large language model has visual relevance with the input image sequence to be processed and instruction text, maintaining the coherence of the plot. The narrative content is consistent with the timeline of the image sequence, achieving high-quality visual narrative. This device can be widely applied in the fields of intelligent content creation, virtual narrative generation, and multimodal artificial intelligence.

[0188] In addition, the device is equipped with a user interface for receiving input sequences of images to be processed and outputting generated visual narrative text.

[0189] The visual narrative generation device based on visual language alignment in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific device.

[0190] The visual narrative generation device based on visual language alignment in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0191] The visual narrative generation apparatus based on visual language alignment provided in this application can realize the various processes implemented in the visual narrative generation method embodiment based on visual language alignment in the above embodiments. To avoid repetition, it will not be described again here.

[0192] In some embodiments, such as Figure 9 As shown, this application embodiment also provides an electronic device 900, including a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901. When the program is executed by the processor 901, it implements the various processes of the above-described visual narrative generation method embodiment based on visual language alignment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0193] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0194] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described visual narrative generation method embodiment based on visual language alignment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0195] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0196] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described visual narrative generation method based on visual language alignment.

[0197] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0198] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described visual narrative generation method based on visual language alignment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0199] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0200] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0201] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the visual narrative generation method based on visual language alignment of the various embodiments of this application.

[0202] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0203] In the description of this application, "multiple" means two or more.

[0204] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0205] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0206] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A visual narrative generation method based on visual language alignment, characterized in that, include: Obtain the image sequence to be processed and the instruction text; The image sequence to be processed and the instruction text are input into the multimodal large model to obtain the visual narrative text output by the multimodal large model; The multimodal large model includes a visual encoder, a multimodal mapper, and a large language model connected in sequence, and also includes a text encoder, the output of which is connected to the input of the large language model; The step of inputting the image sequence to be processed and the instruction text into a multimodal large model to obtain the visual narrative text output by the multimodal large model includes: The visual encoder sequentially encodes the images in the image sequence to be processed to generate visual features. The visual features are sequentially projected into the word embedding space of the large language model through the multimodal mapper for semantic alignment, resulting in language embedding vectors. The text encoder converts the instruction text into an instruction embedding vector. The visual narrative text is generated by performing logical reasoning based on the language embedding vector and the instruction embedding vector using the large language model. Before inputting the image sequence to be processed and the instruction text into the multimodal large model, the method further includes: Obtain multiple image sequence samples from the sample set, and an initial storyline for the multiple image sequence samples, wherein the image sequence samples include multiple sample images; Generate an overall description for each of the sample images in the image sequence samples; Each of the sample images is divided into local regions and a corresponding local description is generated for each. Based on the overall description and the local description, an image description of the sample image is obtained; Based on the initial storyline of the image sequence samples and the image description of each sample image in the image sequence samples, the image theme of the image sequence samples is determined; The initial storyline, the image description, and the image theme are merged to generate story narrative tags for the image sequence samples; The combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text is used as training samples and input into the multimodal large model to train the multimodal large model.

2. The visual narrative generation method based on visual language alignment according to claim 1, characterized in that, Based on the initial storyline of the image sequence samples and the image description of each sample image in the image sequence samples, the image theme of the image sequence samples is determined, including: Based on the image description and the initial storyline, a first theme is generated for each image sequence sample; Based on the data source of the image sequence samples, the first topic is divided into at least one topic set; The first topic in the topic set is deduplicated and filtered to obtain at least one second topic; Based on the initial storyline, the image theme of each image sequence sample is determined by matching the second theme with the corresponding image sequence samples from the data source.

3. The visual narrative generation method based on visual language alignment according to claim 2, characterized in that, Based on the initial storyline, the image theme of each image sequence sample is determined by matching the second theme with the corresponding image sequence samples from the data source, including: The plurality of image sequence samples are returned to the sample set; The image sequences in the sample set are divided according to their data sources; By matching the initial storyline of the image sequence samples with the second theme corresponding to the data source, the image theme of the image sequence samples is determined. The image subject is generated based on the initial storyline and the image description of the unmatched image sequence samples.

4. The visual narrative generation method based on visual language alignment according to claim 1, characterized in that, Before inputting the combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text as training samples into the multimodal large model to train the multimodal large model, the method further includes: Acquire multiple image-text pairs, wherein each image-text pair includes an image file and a corresponding image description text; The image-text pair is input into the multimodal large model to train the modality mapper in the multimodal large model.

5. The visual narrative generation method based on visual language alignment according to claim 1, characterized in that, The combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text is used as training samples and input into the multimodal large model to train the multimodal large model, including: Using the training samples, the probability of the multimodal large model generating story narratives is determined: ； in, It is the output of the multimodal large model based on the k-th training sample; in the sample instruction text, It is the sample instruction text specifying the output content. It is generated Style type; It is generated Word count requirement; It is the input image sequence sample, which includes m sample images. It is the j-th sample image in the image sequence sample; The parameters of the large language model in the multimodal large model are fine-tuned based on the probability.

6. The visual narrative generation method based on visual language alignment according to claim 1, characterized in that, After inputting the combination of the image sequence samples, the corresponding story narrative tags, and the sample instruction text as training samples into the multimodal large model to train the multimodal large model, the method further includes: The training samples are cyclically input into the multimodal large model to obtain multiple narrative outputs from the multimodal large model; The sample images in the training samples are reordered and input into the multimodal large model to obtain multiple narrative outputs from the multimodal large model; The multiple narrative outputs are scored and ranked, and the highest-ranked narrative output is used as a positive example of the image sequence samples in the training samples, while the lowest-ranked narrative output is used as a negative example of the image sequence samples in the training samples. The multimodal large model is optimized by using the training samples to correspond to the positive and negative examples.

7. A visual narrative generation apparatus based on visual language alignment, employing the visual narrative generation method based on visual language alignment as described in any one of claims 1-6, characterized in that, include: The acquisition module is used to acquire the image sequence to be processed and the instruction text; The processing module is used to input the image sequence to be processed and the instruction text into the multimodal large model to obtain the visual narrative text output by the multimodal large model; The multimodal large model includes a visual encoder, a multimodal mapper, and a large language model connected in sequence, and also includes a text encoder, the output of which is connected to the input of the large language model; The processing module is further used for: The visual encoder sequentially encodes the images in the image sequence to be processed to generate visual features. The visual features are sequentially projected into the word embedding space of the large language model through the multimodal mapper for semantic alignment, resulting in language embedding vectors. The text encoder converts the instruction text into an instruction embedding vector. The visual narrative text is generated by performing logical reasoning based on the language embedding vector and the instruction embedding vector using the large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the visual narrative generation method based on visual language alignment as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the visual narrative generation method based on visual language alignment as described in any one of claims 1-6.

Images