Text, image and video multi-modal fusion text generation method and system

Abstract

The invention belongs to the technical field of network space cognitive domains, and discloses a text generation method and system for text, image and video multi-modal fusion. According to the method, on the basis of images, videos and text contents published by a user, important features in multiple modes are extracted through a large model, related descriptions are generated, description information sharing is promoted, key information and core concepts are extracted to achieve deep integration among different modes, and finally an accurate text conforming to the context is generated on the basis of the large model. According to the method, multi-modal fusion is realized, clear and accurate visual information is provided, the problem of single text generation content angle is solved, and the problem of expression fuzziness in text generation is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of cyberspace cognitive domain, and in particular relates to a text generation method and system for multimodal fusion of text, image and video. Background Art

[0002] In terms of text generation, related technologies include image description technology, video understanding technology, and text generation technology. When generating text based on the information provided by text materials, it usually only considers the logical level of the text context semantics in the material, and lacks the integration of knowledge feature information such as images and videos in the original material, resulting in the generated content being off-topic or incoherent. In traditional text generation, the material angles or themes that can be produced based solely on the information provided by the material text are limited.

[0003] Through the above analysis, the problems and defects of the existing technology are as follows: the existing technology generates text from a single angle and the expression in text generation is vague. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide a method and system for generating text by multimodal fusion of text, image and video, and specifically relate to a personalized text generation technology by multimodal fusion of text, image and video. The purpose of the present invention is to provide an additional source of information by combining images and videos, analyzing entities, scenes and emotions in the images, so that the generated text can contain more details and descriptions, thereby improving the content richness and diversity of the text.

[0005] The technical solution is as follows: a text generation method for multimodal fusion of text, image and video, based on the images, videos and text content posted by users, extracts features in multimodality through a large model, generates corresponding descriptions and promotes sharing of description information, extracts key information and core concepts, realizes deep integration between different modalities, and generates accurate text that conforms to the context based on the large model; specifically, the following steps are included:

[0006] S1, text content extraction: Use the big model to obtain unstructured text information from social media, establish three different types of prompts, guide and promote the big model to extract meaningful text description information fragments from text data;

[0007] S2, image content extraction: Unstructured image information obtained from social media is combined with video key frames, and a large model is used to understand the global information and details of the image, establish a series of accurate prompts, and conduct in-depth analysis of the image description information;

[0008] S3, video content extraction: Understand and process social media video content, deeply analyze visual content, use large models to analyze the background and environmental features in video materials according to different prompt requirements, and deeply analyze the verbal communication presented in the video to clarify the emotional color and attitude tendency of the interlocutor; identify whether the information sharing pool content is involved, and analyze the potential connection and intersection between the video content and the information sharing pool content;

[0009] S4, personalized text content generation: The extracted key information and core concept understanding fusion features are input into the large model to generate text output that conforms to the context logic, is semantically rich and highly relevant.

[0010] In step S1, the large model extracts entities, relationships and events according to the three types of prompts, generates detailed text descriptions, and communicates these text description information in the process of information sharing; the entities include identifying meaningful entities, the relationships include identifying the relationships between entities in the text, and the events include identifying events and their related participants, time, and location information.

[0011] In step S2, the image description information is deeply analyzed, including:

[0012] The first part, general understanding, covers the skills of extracting and interpreting information from complex documents in images; accurately identifying and classifying various entities in the image, and marking the specific location coordinates of each entity in the image space;

[0013] The second part is the interpretation of the universal knowledge carried by the images, the analysis of the implicit emotional expressions, and the visual symbols that contain humor and cultural resonance;

[0014] The third part is to mine the potential meaning and related content that echoes the shared information in the text. The content of the first and second parts is extracted in a prompt manner. The third part retrieves the entities involved in the text content from the established information sharing pool, and then marks the location of the entities in the image.

[0015] The above three parts are combined to construct an image description that covers the location associations and contexts between entities in the image and is shared in the information sharing pool.

[0016] In step S2, image content extraction includes:

[0017] S201, adjusting the image size, normalizing the color, and enhancing the image to convert it into a format suitable for the large model;

[0018] S202, using ViT to divide the image into multiple patches, and processing the above patches through a multi-layer self-attention mechanism to form a feature map containing global and local information; introducing position encoding to enhance the position perception ability of each patch feature through a position-aware visual language adapter to understand the relationship between different parts of the image;

[0019] S203, encodes the input prompt text, integrates visual and text features into a unified embedding space through a cross-attention mechanism, and promotes deep interaction;

[0020] S204, the fused features are further processed by a visual decoder to generate an image description output.

[0021] In step S202, the vision-language adapter contains a randomly initialized single-layer cross-attention module, uses a set of trainable vectors as query vectors, and takes image features from the visual encoder as keys for the cross-attention operation.

[0022] In step S204, the image description output is generated, including: integrating visual and text features into a unified embedding space through a cross-attention mechanism to promote deep interaction; generating a description, answer or explanation of the image, including a clear description of the location of the object.

[0023] In step S3, the background and environmental features in the video material include the unique attributes of the scene's spatial structure, meteorological conditions, and geographical environment; it also includes a detailed description of the character's movements and activities, including body movements, facial micro-expressions, and communication with surrounding objects, peers, or other creatures.

[0024] In step S3, video content extraction includes:

[0025] S301, generating a detailed video description based on the first frame of the video content, using a semantically-aware key frame extraction method to capture key frame changes and remove adjacent frames with high semantic similarity;

[0026] S302, applies a differential sliding window in time sequence to process subsequent supporting frames, and takes all differentiated descriptions and their corresponding key frames as input by combining the differential descriptions of the current key frame and its previous frame, thereby capturing the overall narrative clues and detail changes in the video, and superimposing the text description, image description and object positioning content in the information sharing pool to guide the large model to produce a detailed and temporal overall description.

[0027] In step S4, the key information and core concepts extracted include:

[0028] Based on the text description, image description and video description features that are fused at the semantic level across all modalities, we ensure that the information of each modality is accurately captured and expressed. We then use a large model to analyze the meaning of the fused features, further refine deeper meaning associations, understand these fused features, and extract key information and core concepts.

[0029] Another object of the present invention is to provide a text generation system for multimodal fusion of text, image and video, which implements the text generation method for multimodal fusion of text, image and video, and the system comprises:

[0030] The text content extraction module is used to apply the big model to the unstructured text information obtained from social media, establish three different types of prompts, and guide and promote the big model to accurately extract meaningful text description information fragments from the text data;

[0031] Image content extraction module is used to obtain unstructured image information from social media, combine it with video key frames, use a large model to understand the global information and details of the image, establish a series of accurate prompts, and conduct in-depth analysis of image description information;

[0032] The video content extraction module is used to understand and process social media video content, deeply analyze visual content, use large models to analyze the background and environmental features in video materials according to different prompt requirements, and deeply analyze the verbal communication presented in the video to clarify the emotional color and attitude tendency of the interlocutor; identify whether the information sharing pool content is involved, and analyze the potential connection and intersection between the video content and the information sharing pool content;

[0033] The personalized text content generation module is used to input the extracted key information and core concept understanding fusion features into the large model to generate text output that conforms to the context logic, is semantically rich and highly relevant.

[0034] In combination with all the above-mentioned technical solutions, the beneficial effects of the present invention are as follows: based on the problems existing in the existing text generation methods, the present invention proposes a personalized text generation method for multimodal fusion in social media, which realizes multimodal fusion, provides clear and accurate visual information, solves the problem of single content angle in text generation, and reduces the problem of expression ambiguity in text generation.

[0035] On some platforms, the quantity and quality of interactions may affect the ranking of search results and increase product exposure. Previously, personalized text generation was only performed for the text content of the copy. The present invention adds visual information such as pictures and videos, which can generate higher quality text generation after supplementation. Different types of content (text, pictures, videos) are seamlessly combined to form a unified whole. Through diversified data sources, it is ensured that personalized text generation reflects the views and emotions of different users, rather than a single perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0037] Figure 1 It is a flow chart of a text generation method of multimodal fusion of text, image and video provided by an embodiment of the present invention;

[0038] Figure 2 is an overall calculation flow chart of image description provided by an embodiment of the present invention;

[0039] Figure 3 is an overall calculation flow chart of the video description provided by an embodiment of the present invention;

[0040] Figure 4 It is a principle diagram of a text generation method for multimodal fusion of text, image and video provided by an embodiment of the present invention;

[0041] Figure 5 Schematic diagram of a text generation system for multimodal fusion of text, image and video provided by an embodiment of the present invention;

[0042] In the figure: 1. Text content extraction module; 2. Image content extraction module; 3. Video content extraction module; 4. Personalized text content generation module. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0044] The present invention innovatively maps multimodal data of different modalities (text, image, video) into a shared embedding space, enabling large models to exchange and fuse information between different modalities.

[0045] Embodiment 1, as Figure 1 As shown, the text generation method for multimodal fusion of text, image and video provided by the embodiment of the present invention is based on the images, videos and text content posted by users, extracts important features in multimodality through a large model, generates relevant descriptions and promotes effective sharing of description information, extracts key information and core concepts to achieve deep integration between different modalities, and finally generates accurate text that conforms to the context based on this large model. Specifically, it includes the following steps:

[0046] S1, text content extraction: Use the big model to obtain unstructured text information from social media, establish three different types of prompts, guide and promote the big model to extract meaningful text description information fragments from text data;

[0047] Large models usually need to extract three types of prompts: 1. Entities (identify meaningful entities, such as names of people, places, organizations, dates, etc.); 2. Relationships (identify the relationships between entities in the text); 3. Events (identify events and their related participants, time, location, etc.). This process aims to generate detailed text descriptions and ensure that these text descriptions can be conveyed in the process of information sharing.

[0048] S2, image content extraction: Unstructured image information obtained from social media is combined with video key frames, and a large model is used to understand the global information and details of the image, establish a series of accurate prompts, and conduct in-depth analysis of the image description information;

[0049] The analyzed content includes three core parts: the first part is conventional understanding; the second part is the interpretation of the universal cognitive knowledge carried by the image; the third part is to explore the potential meaning and related content that echoes the information shared by the text;

[0050] The first part of the general understanding covers the skills of extracting and interpreting information from complex documents in images; accurately identifying and classifying various entities in the image, and marking the specific location coordinates of each entity in the image space;

[0051] The second part explains the common cognitive knowledge carried by the images, analyzing the implicit emotional expressions, and the visual symbols that contain humor and cultural resonance, such as the understanding of the deep meaning of emoticons;

[0052] The third part mines the potential meaning and related content that echoes the shared information in the text. The content of the first two key parts is extracted using a carefully designed prompt method, while the third part retrieves the entities involved in the text content from the established information sharing pool and then marks the location of the entities in the image.

[0053] The above three parts are combined to construct a detailed description that covers the location associations and contexts between entities in the image, and the image description information is shared in the information sharing pool.

[0054] The specific implementation process is:

[0055] S201, adjusting the image size, normalizing the color, and performing image enhancement to convert the image into a format suitable for the large model.

[0056] S202, using ViT to cut the image into fixed-size patches from top to bottom, and adding position encoding to each patch through multi-layer self-attention mechanism processing, so as to retain the spatial information of the image while extracting features, forming a feature map containing global and local information. The global features describe the overall features of the entire image, such as color distribution, texture and shape. The introduction of position encoding enhances the position perception ability of each patch feature through a position-aware visual language adapter. By combining the position encoding with the patch feature (for example, by addition or concatenation), the feature representation of each patch contains not only its own visual information, but also its relative or absolute position information in the image. Understand the positional relationship between different parts of the image.

[0057] S203, encode the input prompt text, integrate visual and text features into a unified embedding space through the cross-attention mechanism, and promote deep interaction. Visual features refer to the global and local features in front, and text features refer to the text encoding for information extraction from the prompt; the core of the cross-attention mechanism is to calculate the similarity between the two sequences and assign weights to each element according to the similarity to achieve effective fusion of information. Specifically, it allows one sequence to pay attention to the relevant part of another sequence during generation or processing, thereby obtaining richer contextual information.

[0058] S204, the fused features are further processed by the visual decoder, and the position-sensitive reasoning mechanism is used to enable the model to take into account the specific position of the object when generating a description or answer; it is used to generate an image description output.

[0059] In the multimodal fusion stage, the cross-attention mechanism is used to integrate visual and textual features into a unified embedding space to promote deep interaction, and finally generate a description, answer or explanation of the image, including a clear description of the location of the object.

[0060] The position-aware visual language adapter in step S202 is designed to alleviate the efficiency problem caused by long image feature sequences, and includes a randomly initialized single-layer cross-attention module: a set of trainable vectors (embeddings) are used as query vectors, and image features from the visual encoder are used as keys for the cross-attention operation.

[0061] The overall computing process architecture described in this image is as follows Figure 2 shown.

[0062] S3, video content extraction: Understand and process social media video content, deeply analyze visual content, use large models to analyze the background and environmental features in video materials according to different prompt requirements, and deeply analyze the verbal communication presented in the video to clarify the emotional color and attitude tendency of the interlocutor; identify whether the information sharing pool content is involved, and analyze the potential connection and intersection between the video content and the information sharing pool content;

[0063] For example, it can understand and process social media video content comprehensively, deeply analyze visual content, and use large models to analyze the background and environmental features in video materials in detail according to different prompt requirements. Its scope not only covers the spatial structure of the scene, weather conditions, and unique attributes of the geographical environment; it involves detailed descriptions of character movements and activities, including but not limited to body movements, facial micro-expressions, and communication with surrounding objects, peers or other creatures; in addition, it will conduct in-depth analysis of verbal communication presented in the video to gain insight into the emotional color and attitude tendencies of the interlocutors; identify whether the information sharing pool content is involved, and deeply explore the potential connection and intersection between the video content and the information sharing pool content.

[0064] The specific implementation process is as follows:

[0065] S301, firstly, a detailed video description is generated based on the first frame of the video content, a semantically-aware key frame extraction method is adopted to capture key frame changes and remove adjacent frames with high semantic similarity.

[0066] S302, apply the differential sliding window in time sequence to process subsequent supporting frames, by combining the differential description of the current key frame and its previous frame, and taking all differentiated descriptions and their corresponding key frames as input, so as to capture the overall narrative clues and detail changes in the video, and superimpose the text description, image description and object positioning content in the information sharing pool. For example, the text content in the copy is: "Today's weather is really good, suitable for going out", and the content of the video in the copy includes the content of going out. Combining the above information, you can understand the description of the character's activities today; guide the large model to generate a detailed and time-series overall description. The overall calculation process of the video description is as follows: Figure 3 shown.

[0067] S4, personalized text content generation: The extracted key information and core concept understanding fusion features are input into the large model to generate text output that conforms to the context logic, is semantically rich and highly relevant.

[0068] This process ensures that the information of each modality is accurately captured and expressed based on the features of all modalities fused at the semantic level (including text description, image description, and video description information). The meaning of the fused features is then analyzed through a large model to further refine deeper meaning associations, understand these fused features, and extract key information and core concepts. The extracted key information and core concept fusion features are input into the large model to generate text output that conforms to the context logic, is semantically rich, and is highly relevant.

[0069] like Figure 4 Schematic diagram of a text generation method for multimodal fusion of text, image and video provided in an embodiment of the present invention.

[0070] The above embodiments show that the present invention can be used to generate personalized text content that is relevant to topics and conforms to user contexts on social media platforms.

[0071] Embodiment 2, as Figure 5 As shown, the text generation system for multimodal fusion of text, image and video provided by the embodiment of the present invention includes:

[0072] Text content extraction module 1 is used to use the big model in the unstructured text information obtained from social media, establish three different types of prompts, guide and promote the big model to accurately extract meaningful text description information fragments from text data;

[0073] Image content extraction module 2 is used to obtain unstructured image information from social media, combine it with video key frames, use a large model to understand the global information and details of the image, establish a series of accurate prompts, and conduct in-depth analysis of the image description information;

[0074] Video content extraction module 3 is used to understand and process social media video content, deeply analyze visual content, use large models to analyze the background and environmental characteristics of video materials according to different prompt requirements, and deeply analyze the verbal communication presented in the video to clarify the emotional color and attitude tendency of the interlocutor; identify whether the information sharing pool content is involved, and analyze the potential connection and intersection between the video content and the information sharing pool content;

[0075] The personalized text content generation module 4 is used to input the extracted key information and the comprehension fusion features of the core concepts into the large model to generate text output that conforms to the context logic, is semantically rich and highly relevant.

[0076] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A text generation method for multimodal fusion of text, image and video, characterized in that: This method is based on the images, videos and text content posted by users. It extracts features in multimodality through a large model, generates corresponding descriptions and promotes the sharing of description information, extracts key information and core concepts, realizes deep integration between different modalities, and generates accurate text that conforms to the context based on the large model. Specifically, it includes the following steps: S1, text content extraction: Use the big model to obtain unstructured text information from social media, establish three different types of prompts, guide and promote the big model to extract meaningful text description information fragments from text data; S2, image content extraction: Unstructured image information obtained from social media is combined with video key frames, and a large model is used to understand the global information and details of the image, establish a series of accurate prompts, and conduct in-depth analysis of the image description information; S3, video content extraction: Understand and process social media video content, deeply analyze visual content, use large models to analyze the background and environmental features in video materials according to different prompt requirements, and deeply analyze the verbal communication presented in the video to clarify the emotional color and attitude tendency of the interlocutor; identify whether the information sharing pool content is involved, and analyze the potential connection and intersection between the video content and the information sharing pool content; S4, personalized text content generation: The extracted key information and core concept understanding fusion features are input into the large model to generate text output that conforms to the context logic, is semantically rich and highly relevant.

2. The text generation method of multimodal fusion of text, image and video according to claim 1 is characterized in that: In step S1, the large model extracts entities, relationships and events according to the three types of prompts, generates detailed text descriptions, and communicates these text description information in the process of information sharing; the entities include identifying meaningful entities, the relationships include identifying the relationships between entities in the text, and the events include identifying events and their related participants, time, and location information.

3. The text generation method of multimodal fusion of text, image and video according to claim 1 is characterized in that: In step S2, the image description information is deeply analyzed, including: The first part, general understanding, covers the skills of extracting and interpreting information from complex documents in images; accurately identifying and classifying various entities in the image, and marking the specific location coordinates of each entity in the image space; The second part is the interpretation of the universal knowledge carried by the images, the analysis of the implicit emotional expressions, and the visual symbols that contain humor and cultural resonance; The third part is to mine the potential meaning and related content that echoes the shared information in the text. The content of the first and second parts is extracted in a prompt manner. The third part retrieves the entities involved in the text content from the established information sharing pool, and then marks the location of the entities in the image. The above three parts are combined to construct an image description that covers the location associations and contexts between entities in the image and is shared in the information sharing pool.

4. The text generation method of multimodal fusion of text, image and video according to claim 1 is characterized in that: In step S2, image content extraction includes: S201, adjusting the image size, normalizing the color, and enhancing the image to convert it into a format suitable for the large model; S202, using ViT to divide the image into multiple patches, and processing the above patches through a multi-layer self-attention mechanism to form a feature map containing global and local information; introducing position encoding to enhance the position perception ability of each patch feature through a position-aware visual language adapter to understand the relationship between different parts of the image; S203, encodes the input prompt text, integrates visual and text features into a unified embedding space through a cross-attention mechanism, and promotes deep interaction; S204, the fused features are further processed by a visual decoder to generate an image description output.

5. The text generation method of multimodal fusion of text, image and video according to claim 4 is characterized in that: In step S202, the vision-language adapter contains a randomly initialized single-layer cross-attention module, uses a set of trainable vectors as query vectors, and takes image features from the visual encoder as keys for the cross-attention operation.

6. The text generation method of multimodal fusion of text, image and video according to claim 4 is characterized in that: In step S204, the image description output is generated, including: integrating visual and text features into a unified embedding space through a cross-attention mechanism to promote deep interaction; generating a description, answer or explanation of the image, including a clear description of the location of the object.

7. The text generation method of multimodal fusion of text, image and video according to claim 1 is characterized in that: In step S3, the background and environmental features in the video material include the unique attributes of the scene's spatial structure, meteorological conditions, and geographical environment; it also includes a detailed description of the character's movements and activities, including body movements, facial micro-expressions, and communication with surrounding objects, peers, or other creatures.

8. The text generation method of multimodal fusion of text, image and video according to claim 1, characterized in that: In step S3, video content extraction includes: S301, generating a detailed video description based on the first frame of the video content, using a semantically-aware key frame extraction method to capture key frame changes and remove adjacent frames with high semantic similarity; S302, applies a differential sliding window in time sequence to process subsequent supporting frames, and takes all differentiated descriptions and their corresponding key frames as input by combining the differential descriptions of the current key frame and its previous frame, thereby capturing the overall narrative clues and detail changes in the video, and superimposing the text description, image description and object positioning content in the information sharing pool to guide the large model to produce a detailed and temporal overall description.

9. The text generation method of multimodal fusion of text, image and video according to claim 1, characterized in that: In step S4, the key information and core concepts extracted include: Based on the text description, image description and video description features that are fused at the semantic level across all modalities, we ensure that the information of each modality is accurately captured and expressed. We then use a large model to analyze the meaning of the fused features, further refine deeper meaning associations, understand these fused features, and extract key information and core concepts.

10. A text generation system for multimodal fusion of text, image and video, characterized in that: The method for generating text by multimodal fusion of text, image and video according to any one of claims 1 to 9 is implemented, and the system comprises: The text content extraction module (1) is used to use the big model to extract unstructured text information obtained from social media, establish three different types of prompts, and guide and promote the big model to accurately extract text description information fragments with substantive meaning from the text data; Image content extraction module (2), which is used to obtain unstructured image information from social media, combine it with video key frames, use a large model to understand the global information and details of the image, establish a series of accurate prompts, and conduct in-depth analysis of the image description information; The video content extraction module (3) is used to understand and process the social media video content, deeply analyze the visual content, use a large model to analyze the background and environmental characteristics of the video material according to different prompt requirements, and deeply analyze the verbal communication presented in the video to clarify the emotional color and attitude tendency of the interlocutor; identify whether the information sharing pool content is involved, and analyze the potential connection and intersection between the video content and the information sharing pool content; The personalized text content generation module (4) is used to input the extracted key information and the comprehension fusion features of the core concepts into the large model to generate text output that conforms to the context logic, is semantically rich and highly relevant.

Images