Chat system and method based on visual content dialogue
By combining a visual feature encoding module with a large language model, the problem of multimodal question answering models being concentrated in specific scenarios in the training dataset was solved, realizing multimodal question answering in open-domain videos and improving the model's generalization ability and dialogue accuracy.
Patent Information
- Application Number
- PCT/CN2024/095036
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-05-23
- Publication Date
- 2024-11-14
AI Technical Summary
Existing multimodal question answering models are trained on specific scenarios, resulting in poor performance in the real world. They suffer from gaze bias and insufficient fusion of visual and audio information, making it difficult to resolve complex questions.
The system employs a visual feature encoding module and a large language model to generate visually encoded content through visual feature encoding, and combines this with user text input to generate dialogue output. It uses a progressive training framework and multimodal instruction fine-tuning of data collection methods to improve the model's generalization ability and accuracy.
It improves the generalization ability and accuracy of the multimodal question answering system, enabling multimodal question answering in open-domain videos, capturing video temporal information and generating accurate dialogue output.
Smart Images

Figure CN2024095036_14112024_PF_FP_ABST
Abstract
Description
A chat system and method based on visual content dialogue Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and more particularly, to a chat system and method for conducting conversations based on visual content. Background Art
[0002] Multimodal question answering models are designed to answer a question based on the visual content of an image or video clip. Existing approaches to this problem all use generative models, including text-based, vision-based, audio-based, and multimodal approaches.
[0003] Specifically, text-based methods mainly rely on analyzing the text in the video to answer questions related to the video. For example, subtitles or text appearing in the scene are detected and recognized through OCR technology. Vision-based methods focus on analyzing the visual content of the video, such as objects, scenes, actions, etc., in order to extract key information from them. Typical visual features include features extracted by CNN (convolutional neural network) or pre-trained models (such as ViT or SwAV). Audio-based methods analyze the audio signals in the video to answer questions related to sound. For example, identifying specific people, music types or background noises through sound. Multimodal methods integrate multiple information sources such as text, vision and audio to solve video question answering tasks. This usually includes feature extraction and representation learning for each modality, and combining all information through fusion algorithms.
[0004] Video question answering is a method for evaluating a computer's ability to understand video content. Analysis of existing solutions reveals the following main flaws:
[0005] 1) Data bias: Current model training datasets are overly focused on certain scenarios or fields, resulting in poor performance of the models in the real world.
[0006] 2) Model gaze shift: The model fails to focus on all important areas related to the question, resulting in inaccurate answers.
[0007] 3) In multimodal methods, effectively fusing visual information in videos with audio information remains challenging, which may lead to insufficient fusion or overfitting, and the complex question structure may make it difficult for the model to parse the question and produce the correct answer.
[0008] There is a need in the art for a chat system and method for conducting conversations based on visual content that is improved in one or more of the above aspects.
[0009] Summary of the Invention
[0010] In order to overcome at least one of the above-mentioned deficiencies in the prior art, a chat system and method for conducting conversations based on visual content are provided.
[0011] According to a first aspect of the present invention, a chat system for conducting conversations based on visual content is provided. The chat system includes a visual feature encoding module and a large language model. The visual feature encoding module is configured to generate visual feature encodings corresponding to visual inputs, the visual feature encodings being used to form visually encoded content. The large language model is configured to generate corresponding conversation outputs based on the visually encoded content and user text input.
[0012] According to a second aspect of the present invention, a chat method for conducting a conversation based on visual content is provided. The method comprises the following steps: generating, by a visual feature encoding module, a visual feature encoding corresponding to a visual input, the visual feature encoding being used to form visually encoded content; and generating, by a large language model, a corresponding conversation output based on the visually encoded content and user text input.
[0013] The chat system and chat method for conducting conversation based on visual content according to the embodiments of the present invention have strong generalization capability and high accuracy.
[0014] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0016] FIG1 is a block diagram of a chat system based on visual content according to an embodiment of the present invention.
[0017] FIG2 is a schematic diagram of a chat system for communicating based on visual content according to an embodiment of the present invention.
[0018] FIG3 is a schematic diagram of a process of a video pre-training model according to an embodiment of the present invention.
[0019] FIG4 is a schematic diagram of a process of a video pre-training model according to an embodiment of the present invention.
[0020] FIG5 is a flowchart of a method for chatting based on visual content according to an embodiment of the present invention.
[0021] 6 is a flow diagram of a process for training a large language model according to some embodiments of the invention.
[0022] FIG7 is a schematic diagram showing the display effect of a chat system based on visual content dialogue according to an embodiment of the present invention.
[0023] FIG8 is a schematic diagram showing the display effect of a chat system based on visual content dialogue according to an embodiment of the present invention. DETAILED DESCRIPTION
[0024] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.
[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0026] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0027] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0028] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0029] According to one aspect of the present invention, a chat system based on visual content is provided.
[0030] Figure 1 is a block diagram of a visual content-based chat system 100 according to one embodiment of the present invention. Chat system 100 has a wide range of applications, including chatbots, real-time barrage Q&A, tools to help the blind watch videos, security Q&A robots, and entertainment Q&A robots.
[0031] The chat system 100 may include a visual feature encoding module 110 and a large language model 120. The visual feature encoding module 110 may be configured to receive visual input 130 and generate a visual feature encoding corresponding to the visual input 130. The visual feature encoding may be used to form visually encoded content. For example, the visual input 130 may include a video, an image, etc. The large language model may be configured to generate corresponding conversation output based on the visually encoded content and user text input.
[0032] By forming visual encoding content including visual feature encoding and using it as input to a large language model, the chat system according to an embodiment of the present invention is more accurate than text-based implementations, can make up for the shortcomings of visual content description text obtained by traditional perception methods in understanding complex temporal reasoning and causal inference, and is better compatible with visual content embedding in multimodal dialogue tasks.
[0033] In some embodiments, the chat system 100 may further include a perception module 140 . The perception module 140 may be configured to convert the visual input 130 into structured visual content description text, which is also used to form visual encoding content as one of the inputs of the large language model 120 .
[0034] In some embodiments, the visual feature encoding module 110 may include a video encoder 112 and a code mapping module 114. In some embodiments, the visual encoder 112 may include an image encoder (e.g., CLIP ViT-G) and a global multi-head relation aggregator (GMHRA). This visual encoder 112 is a powerful temporal modeling module. In this embodiment, all visual tokens of the image encoder are processed and knowledge is fully distilled directly from a pre-trained image model (e.g., CLIP).
[0035] In some embodiments, the visual encoder 112 may include a feature encoder (e.g., UMT ViT-L) containing a spatiotemporal attention mechanism. For this embodiment, the visual tokens of the visual encoder 112 may be masked, and only the non-masked visual tokens are processed, so that knowledge can be partially distilled from a pre-trained image model (e.g., CLIP). Compared with the embodiment using ViT-G, in the upgraded dialogue system, the secondary development method of the image-based model of ViT-G destroys the generalization inherited from the image-based model, and image pre-training easily causes the video model to be too "short-sighted", ignoring the global spatiotemporal relationship, while also limiting the amplification of the model. This implementation introduces a progressive training framework that uses a spatiotemporal joint attention mechanism to process only the non-masked visual tokens in ViT. The features of the teacher and the student are aligned through the MSE loss, so that the student model can distill knowledge from the pre-trained image model (e.g., CLIP).
[0036] In some embodiments, the encoding-mapping module 114 may include a QFormer. In some embodiments, the encoding-mapping module 114 may include a linear projection module. By adopting a pre-trained QFormer and adding additional linear projections, more query tokens are added to the complex video context. This approach enables the acquisition of compact, pre-trained text-compatible video embeddings to facilitate multimodal question-answering using the chat system 100.
[0037] In some embodiments, the large language model may include a low-rank adaptive module that operates in conjunction with the visual feature encoding module 110 during training to be adjusted by a visually guided text generation loss. This is a class of techniques that aims to reduce the complexity of a large model by approximating its high-dimensional structure with a low-dimensional structure. In the context of a language model, this means creating a smaller, more manageable representation of the original model that can still perform well for a specific task or domain. It combines a visual encoder 112 and an encoding mapping module 114 to adjust the text generation loss through visual guidance. In addition, the instruction data of the visual question and answer pair can be integrated into the encoding mapping module 114 to extract visual tags related to the instruction as input to the large language model 120. The low-rank adaptive module will be further described below in conjunction with Figure 4.
[0038] In some embodiments, the large language model can be trained by fine-tuning using multiple visual-text pairs to obtain fine-tuning results; and fine-tuning instructions based on video multimodal instruction fine-tuning data, which includes visual input and question-answering data collected from the large language model. This process is further described below in conjunction with Figure 3.
[0039] In some embodiments, the large language model can be trained by performing preliminary fine-tuning based on multiple visual-text pairs to obtain preliminary fine-tuning results; performing interface fine-tuning based on video multimodal interface fine-tuning data to obtain interface fine-tuning results, wherein the video multimodal interface fine-tuning data includes visual input and question-answering data collected from the large language model; and performing instruction fine-tuning based on video multimodal instruction fine-tuning data, wherein the video multimodal interface instruction data includes visual input and question-answering data with instructions collected from the large language model. This process will be further described below in conjunction with Figure 4.
[0040] Figure 2 is a schematic diagram of a chat system for conducting conversations based on visual content according to one embodiment of the present invention. The chat system shown in Figure 2 can be an implementation of the chat system 100 described above in conjunction with Figure 1, but the scope of the present invention is not limited thereto. As shown in Figure 2, the present invention is a chat tool for conducting open-domain conversations based on video content. The provided solution is mainly divided into two parts. First, a video-centric multimodal instruction fine-tuning data collection method is designed for the first time. This data collection method can be used to collect detailed text description pairs associated with video images and generate dense captions arranged in chronological order. The data collection process emphasizes spatiotemporal objects, actions, events, and causal relationships, providing valuable resources for training and evaluating video-centric multimodal conversation systems. Second, a new and effective video-centric multimodal conversation system is developed. The system involves two modules: a perception module that converts videos into structured video description text for direct input into a text-based conversation system; and a video pre-training module that aligns visual features with text features of a text-based conversation system for use in a text-based conversation system. Together, these modules form a multimodal conversation system capable of supporting the widespread application of multimodal conversation systems in various fields.
[0041] Some embodiments provide a video-centric instruction fine-tuning data collection method: In this instruction fine-tuning, the corresponding detailed descriptions and question-and-answer data are generated by a pre-trained natural language model based on a video perception module (described in detail below) and some prompts regarding spatiotemporal features. As an example, the perception module can be perception module 140 of the chat system 100 described in conjunction with FIG1 . Compared to video detailed descriptions, video question-answer pairs are introduced to further improve the diversity, temporality, and causal characteristics of the video instruction data.
[0042] In one embodiment of the video description data collection method, a pre-trained natural language model is used to modify the provided video description into a video narrative. This highlights the temporal aspect of the video by depicting its development over time. The first prompt word transforms the various predicted text labels into a coherent, evolving story, as follows:
[0043] Here's a video about {origin_caption}. The video's contents, in chronological order, are: {video_description}. Describe the video in detail, using the adverbs of sequence "first," "next," "then," and "finally," but avoid mentioning specific times. Provide as much detail as possible. Describe everything you saw. The description should be no more than 150 words but no more than 200.
[0044] The second refines the narrative to improve clarity and coherence while reducing the amount of fluff, as follows:
[0045] Please correct the errors in the given paragraph. Delete any duplicate sentences, meaningless characters, non-English sentences, etc. Remove unnecessary repetitions. Rewrite any incomplete sentences. Return the result without explanation. If the input paragraph is already correct, return it without explanation.
[0046] Video Question-Answer Pairs: With video descriptions, we used a pre-trained large natural language model to generate multi-turn conversations for three types of prompts: descriptive, temporal, and causal. For the temporal and causal parts, we proposed a prompt table as shown below, focusing on temporal perception / reasoning and explaining / revealing intent / reasoning.
[0047] As an AI visual assistant, you are observing a video. A description of the video is presented to you in chronological order, detailing the types of objects, their locations (using coordinates), properties, interactions between objects, behaviors, and the environment. Based on this description, you need to answer all questions as if you were watching the video directly. Create a dialogue that simulates you as the visual assistant observing the video and answering questions about the video content. Make sure your answers reflect the tone of a visual AI assistant actively participating in observing the video and answering questions. Include different queries and corresponding responses.
[0048] Covers questions related to the visual content of the video, such as object types, attributes, object counts, behaviors, locations, relative positions of objects, changes in object behavior or positions over time, and interactions between objects. Includes only questions with clear answers: Questions that can be confidently observed and answered based on the video content Questions that can be confidently determined to be unrelated to the video
[0049] Next, include questions related to time perception and reasoning, such as asking what someone did before and after an event, or asking for the specific timestamps of specific events or actions.
[0050] Additionally, complex questions related to the video content are included, such as asking for background knowledge about objects or behaviors in the video, discussing events that occur in the video, exploring hypothetical topics in depth (e.g., what might happen if a person lost their phone while playing with it in the video), explaining a character's emotions or behavior based on their experiences in the video, or predicting how the video's story or scene will develop.
[0051] Because you receive video descriptions as you watch videos, prioritize questions about visual changes over time and the reasons or causes behind those changes, rather than inferring them from a single frame.
[0052] Be careful not to ask for details you're unsure about. When answering complex questions, provide detailed answers, incorporating examples or reasoning to make them more convincing and structured. Use multiple paragraphs if necessary. If a question can't be answered based on the given description, respond with "The provided video doesn't provide this information" rather than indicating that the information comes from the text description.
[0053] Some embodiments unify video-related tasks into a multi-round video question answering format, consisting of an explicit structured text module for generating video descriptions and a video pre-trained encoding and mapping module for projecting visual feature encodings.
[0054] In some embodiments, tasks are defined by text in real-time reasoning, with few or no given learning examples. In this formulation, the present invention treats a large pre-trained text model as a general-purpose video task decoder, converting video-related descriptions or embeddings into human-understandable text. This process is very user-friendly, allowing the base model to be used across a wide range of applications.
[0055] In one embodiment, concepts are extracted from videos using a visual model as follows: where E represents a text description or embedding according to the context, Denotes the jth image model to predict human-readable annotations or visual features, while I and V denote images and videos, respectively. The decoding task prediction is then performed from the text pre-trained large model according to the user’s question as follows:
[0056] in, and They represent the answers of the w text pre-trained large model in round t and all questions given by users before round t. 11 m represents a large text pre-training model.
[0057] From a technical perspective, an ideal end-to-end video multimodal dialogue system should use a video / vision base model (encoder) to convert visual sequences into latent features of a large text pre-trained model to ensure the overall differentiability of the system.
[0058] In one embodiment, the perception module uses multiple vision models to convert visual data from the video into text. Subsequently, specially designed prompts are created to temporally structure the predicted text. Finally, a pre-trained large text model is used to process user-specified tasks by inputting questions based on the text description of the video.
[0059] Specifically, for a given video, a video frame extraction tool is used to extract key frames from the video at a specific frame rate, resulting in T video frames and associated audio. Furthermore, by feeding the extracted frames and audio into various models, action labels, frame summaries, video labels, dense object descriptions, object location coordinates, video subtitles, timestamps, and other details related to the clip are obtained. The relevant content is then integrated into the subtitles to generate a timestamped text description of the video.
[0060] For example, the prompt words are as follows:
[0061] ****************************
[0062] Among them, our prompt words are as follows:
[0063] You are a chatbot that conducts conversations based on video scenes. You primarily answer questions based on the given context, modify the content based on tag information, and provide information about the people or objects in the video. Temporal descriptions are given every few seconds, so you can convert them into time. When describing, please primarily refer to temporal descriptions. Dense captions provide content every five seconds, allowing you to disambiguate between them temporally. However, please do not create a video plot out of thin air. Let's get started!
[0064] Chronological video context: {video segment classification labels, video subtitles
[0065] Timestamp 1 segment title, segment tag, dense subtitle, video subtitle...
[0066] Timestamp 1 segment title, segment tag, dense subtitle, video subtitle...
[0067] Timestamp 2 segment title, segment label, dense subtitle, video subtitle...
[0068] }
[0069] Question: {question}
[0070] **********************************
[0071] Previously, the effectiveness of large pre-trained text models as general video task interpreters was demonstrated through the perceptual coding module proposed in this paper. This method uses various visual models to convert videos into text streams, which can then be used to perform discrimination / inference tasks using the pre-trained text model.
[0072] Considering that relying solely on the perception module is still insufficient in understanding complex temporal reasoning and causal inference. Therefore, in one embodiment, a pre-trained model encoding and mapping module is introduced, which is a multimodal system that combines video and language based models. As an example, the pre-trained model encoding and mapping module can be the visual feature encoding module 110 described above in conjunction with Figure 1. By fine-tuning the video guidance data, it greatly improves the performance in high-order temporal tasks. In some embodiments, the module integrates the pre-trained ViT-G with the global multi-head relationship aggregator (GMHRA), which is a powerful temporal modeling module.
[0073] In addition, considering the use of the above-mentioned ViT-G in some embodiments in the upgraded dialogue system, the secondary development method of the image-based model will destroy the generalization inherited from the image-based model, and image pre-training can easily cause the video model to be too "short-sighted", ignoring the global spatiotemporal relationship, while also limiting the amplification of the model. Therefore, in some embodiments, a progressive training framework is provided, which uses a joint spatiotemporal attention mechanism to process only the non-masked visual tokens in ViT. The features of the teacher and the student are aligned through the MSE loss, so that the student model can distill knowledge from the pre-trained image model CLIP.
[0074] In addition, for the token interface, we adopt the pre-trained QFormer and add additional linear projections, which add more query tokens for complex video contexts. This approach can obtain compact text pre-trained model compatible with video embedding for subsequent multimodal question answering.
[0075] In some embodiments, training focuses on the newly integrated global multi-head relation aggregator (GMHRA), query tokens, and linear projections, while keeping other components unchanged. In some embodiments, training focuses on the newly integrated feature encoder with spatiotemporal attention mechanism, query tokens, and linear projections, while keeping other components unchanged. First, large-scale video-to-text fine-tuning is used to align the modified video encoder with the text pre-trained model. Then, two types of videos are used to train the dialogue system: detailed video descriptions and video question-answer pairs.
[0076] Figure 3 is a schematic diagram of the process of video pre-training model according to one embodiment of the present invention. The process can be divided into the following two stages:
[0077] Phase 1: Alignment. To strike a balance between training convergence and efficiency, 25 million visual-text pairs are introduced for a round of fine-tuning.
[0078] Phase 2: Instruction fine-tuning, which further trains the fine-tuning results from Phase 1. This phase uses question-answering data collected by the data collection system for fine-tuning to improve the model's generalization capabilities. It also incorporates guidance from human expressions to help the model better understand and generate human language.
[0079] The visual feature encoding module in FIG3 may be the visual encoder 112 described above in conjunction with FIG1 , QFormer may be the encoding mapping module 114 in FIG1 , and the text pre-training model may be the large language model 120 in FIG1 .
[0080] FIG3 also uses snowflake symbols to indicate that the corresponding modules are in a “frozen” state, i.e., the parameters of these modules / models are not updated, and uses flame symbols to indicate that the corresponding modules are in an “open” state, i.e., the parameters of the corresponding modules / models will be updated in the current round of training.
[0081] FIG4 is a schematic diagram of a process of a video pre-training model according to an embodiment of the present invention.
[0082] The process in FIG. 4 may include stage 410 , stage 420 , and stage 450 .
[0083] At stage 410, alignment is performed. To achieve a balance between training convergence and efficiency, preliminary fine-tuning of the visual-text pair is introduced. This preliminary fine-tuning can be performed by a visual encoder 411 and an encoding-mapping module 412. For example, the visual encoder 411 can be the visual encoder 112 described above in conjunction with FIG. 1 , and the encoding-mapping module 412 can be the encoding-mapping module 114 described above in conjunction with FIG. 1 .
[0084] Specifically, the visual encoder 411 may receive a visual input 413 and generate an output K / V. For example, the visual input 413 may include a video and / or an image. The encoding mapping module 412 may receive the output K / V, a query 414, and text 415 to output a result. Based on the result, various losses 416 may be determined, and the training of this stage 410 may be completed by comparing the losses 416 under various parameters. For example, the losses 416 may include one or more of a visual text description loss, a visual text matching loss, and a visually guided text generation loss.
[0085] In stage 430, the visual-language interface is fine-tuned. This stage 430 is to further train the fine-tuning results of stage 410. This stage 430 uses the question-answering data collected by the data collection system and combines it with the visual input for fine-tuning to improve the generalization ability of the model. At the same time, guidance information of human expressions can also be introduced to help the model better understand and generate human language. The visual-language interface fine-tuning can be performed by the visual encoder 431 and the encoding mapping module 432 in combination with the large language model 433. As an example, the visual encoder 431 can be the visual encoder 112 described above in conjunction with Figure 1, the encoding mapping module 432 can be the encoding mapping module 114 described above in conjunction with Figure 1, and the large language model 433 can be the large language model 120 described above in conjunction with Figure 1.
[0086] Specifically, the visual encoder 431 can receive a visual input 434 and generate an output K / V. As an example, the visual input 434 can include a video and / or an image. The encoding mapping module 432 can receive the output K / V and the query 414 to output visual encoding content 436. Based on the visual encoding content 436 and text 437a (user input), text 437b can be output, and the visually guided text generation loss 438 can be determined in this process. The training of this stage 430 can be completed by comparing the visually guided text generation loss 438 under various parameters. In this stage 430, a linear projection module 432a is added to the encoding mapping module 432 so that the generated visual encoding content 436 can be used to train the subsequent large language model 433.
[0087] In stage 450, visual-language instruction fine-tuning is performed. This stage 450 is to further train the results of the fine-tuning in stage 430. This stage uses the instruction fine-tuning dataset converted from the original dataset of different capabilities to perform fine-tuning to improve the model's ability to follow instructions and other capabilities. The visual-language instruction fine-tuning can be performed by the visual encoder 451 and the encoding mapping module 452 in combination with the large language model 453. As an example, the visual encoder 451 can be the visual encoder 112 described above in conjunction with Figure 1, the encoding mapping module 452 can be the encoding mapping module 114 described above in conjunction with Figure 1, and the large language model 453 can be the large language model 120 described above in conjunction with Figure 1.
[0088] Specifically, the visual encoder 451 can receive visual input 454 and generate output K / V. As an example, the visual input 454 can include video and / or image. The encoding mapping module 452 can receive the output K / V, query 435 and instruction 456 (involved in the question-answering dialogue) to output visual encoding content 437. Based on the visual encoding content 457 and question 437a (user input), an answer 458b can be output, and the visually guided text generation loss 459 can be determined in this process. The training of this stage 450 can be completed by comparing the visually guided text generation loss 459 under various parameters. In this stage 450, a linear projection module 452a is added to the encoding mapping module 452 so that the generated visual encoding content 457 can be used for training subsequent large language models 453. In this stage 450, a low-rank adaptive module 453a is also added to the large language model 453, which is intended to reduce its complexity by approximating the high-dimensional structure of a large model with a low-dimensional structure. In the context of language models, this means creating a smaller, more manageable representation of the original model that still performs well for a specific task or domain. The addition of this low-rank adaptive module 453a adds new multimodal model capabilities while preserving the original large language model 453 as much as possible, thereby further improving the chat system's capabilities in video action recognition, object recognition, positional relationships, scenes, optical character recognition, and reasoning.
[0089] According to one aspect of the present invention, a chatting method based on visual content is provided.
[0090] 5 is a flow chart of a chat method 500 based on visual content according to an embodiment of the present invention. The chat method 500 may be executed by the chat system 100 described above in conjunction with FIG1 , but the scope of the present invention is not limited thereto.
[0091] The chatting method 500 may include step 510: generating, by a visual feature encoding module, a visual feature code corresponding to the visual input, wherein the visual feature code is used to form visual encoding content.
[0092] The chatting method 500 may include step 530: generating corresponding dialogue output based on the visual encoding content and the user text input by the large language model.
[0093] Optionally, in some embodiments, the chatting method 500 may further include step 550: converting the visual input into structured visual content description text by the perception module, wherein the visual content description text is also used to form the visual coding content.
[0094] FIG6 is a flowchart of processes 610 and 650 for training a large language model according to some embodiments of the present invention. Processes 610 and 650 may be used to train the large language model 120 described above in conjunction with FIG1 or the large language model involved in the chat method 500 described in conjunction with FIG5 , but the scope of the present invention is not limited thereto.
[0095] The process 610 may include step 612 : performing fine-tuning using a plurality of visual-text pairs to obtain a fine-tuning result.
[0096] Process 610 may include step 614 of performing instruction fine-tuning based on video multimodal instruction fine-tuning data, the video multimodal instruction fine-tuning data including visual input and question-answering data collected from a large language model.
[0097] Further details of process 610 may be as described above in conjunction with FIG. 3 .
[0098] The process 650 may include step 652 : performing preliminary fine-tuning based on a plurality of visual-text pairs to obtain preliminary fine-tuning results.
[0099] Process 650 may include step 654: performing interface fine-tuning based on video multimodal interface fine-tuning data to obtain an interface fine-tuning result, wherein the video multimodal interface fine-tuning data includes visual input and question-answering data collected from a large language model.
[0100] Process 650 may include step 656 of performing instruction fine-tuning based on video multimodal instruction fine-tuning data, the video multimodal interface instruction data including visual input and question-answering data with instructions collected from a large language model.
[0101] Further details of process 650 may be as described above in conjunction with FIG. 4 .
[0102] To further verify the effectiveness of the present invention, a system was launched and can operate normally on the vchat.opengvlab.com system, as shown in Figures 7 and 8. The present invention can be applied to a variety of scenarios, such as chat robots, real-time barrage Q&A, tools to help blind people watch videos, security Q&A robots, and entertainment Q&A robots.
[0103] In summary, the present invention provides a set of methods for collecting video multimodal instruction fine-tuning data, a method for explicitly text-structuring videos for use in text dialogue systems, and a method for implicitly encoding videos for use in building multimodal dialogue systems. Compared to the prior art, the present invention can perform multimodal question-answering on open-domain videos; can capture the timing information of videos and provide answers based on the timing information; can implicitly encode videos to interact directly with text dialogue models, and utilize the powerful text generation capabilities of text dialogue models to improve the accuracy of responses. In short, the present invention is the first to use implicit video encoding in the construction of a multimodal dialogue system, in which the text model can also be replaced with other model structures; and different image description models can also be used to generate visual text descriptions during the explicit encoding process.
[0104] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0105] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.
[0106] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.
[0107] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, Python, and conventional procedural programming languages such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be personalized by utilizing the state information of the computer readable program instructions. The electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.
[0108] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0109] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0111] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.
[0112] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.
Claims
1. A chat system based on visual content dialogue, comprising: A visual feature coding module, used to generate a visual feature code corresponding to the visual input, wherein the visual feature code is used to form a visual coding content; as well as A large language model is used to generate corresponding dialogue output based on the visual encoding content and user text input.
2. The system according to claim 1, characterized in that It also includes a perception module, which is used to convert the visual input into a structured visual content description text, wherein the structured visual content description text is also used to form the visual encoding content.
3. The chat system according to claim 1, characterized in that: The visual feature encoding module includes a visual encoder and a coding mapping module.
4. The chat system according to claim 3, characterized in that: The visual encoder includes an image encoder and a global multi-head relationship aggregator, or the visual encoder includes a feature encoder including a spatiotemporal attention mechanism.
5. The chat system according to claim 4, characterized in that: For the feature encoder with spatiotemporal attention mechanism, only non-masked visual tokens are processed, enabling partial distillation of knowledge from the pre-trained image model.
6. The chat system according to claim 3, characterized in that: The encoding mapping module includes a linear projection module.
7. The chat system according to claim 1, characterized in that: The large language model includes a low-rank adaptation module that operates in conjunction with the visual feature encoding module during training to be adjusted by a visually guided text generation loss.
8. The chat system according to claim 1, characterized in that: The large language model is trained by the following steps: Fine-tuning using multiple visual-text pairs to obtain fine-tuning results; and Instruction fine-tuning is performed based on video multimodal instruction fine-tuning data, wherein the video multimodal instruction fine-tuning data includes the visual input and question-answering data collected from the large language model.
9. The chat system according to claim 1, characterized in that: The large language model is trained by the following steps: Perform preliminary fine-tuning based on multiple visual-text pairs to obtain preliminary fine-tuning results; Performing interface fine-tuning based on video multimodal interface fine-tuning data to obtain an interface fine-tuning result, wherein the video multimodal interface fine-tuning data includes the visual input and the question-answering data collected from the large language model; as well as Instruction fine-tuning is performed based on video multimodal instruction fine-tuning data, wherein the video multimodal interface instruction data includes the visual input and question-answering data with instructions collected from the large language model.
10. The chat system according to claim 8 or 9, characterized in that: The question-answering data is constructed through the following steps: With the goal of describing the temporality of the video, the target video description is modified into a video narrative using the pre-trained first natural language model; improving the video narrative with the goal of improving clarity and coherence to obtain a revised video narrative; Based on the modified video narration, a pre-trained second natural language model is used to generate multiple rounds of dialogue for prompts, wherein the prompts have one or more types of descriptive, temporal, and causal content.
11. The chat system according to claim 1, characterized in that: The prediction output of the large language model is determined in the following way: in, and They represent the answers of the large language model in round t and all questions asked by users before round t, respectively. llm represents the large language model, and E represents the text description.
12. A chat method based on visual content dialogue, comprising the following steps: The visual feature encoding module generates a visual feature code corresponding to the visual input, wherein the visual feature code is used to form a visual encoding content; as well as The large language model generates corresponding dialogue output based on the visual encoding content and user text input.
13. The chatting method according to claim 12, characterized in that: The large language model is trained by the following steps: Utilize multiple visual-text pairs for fine-tuning to obtain fine-tuning results; Instruction fine-tuning is performed based on video multimodal instruction fine-tuning data, wherein the video multimodal instruction fine-tuning data includes the visual input and question-answering data collected from the large language model.
14. The chatting method according to claim 12, characterized in that: The large language model is trained by the following steps: Perform preliminary fine-tuning based on multiple visual-text pairs to obtain preliminary fine-tuning results; Performing interface fine-tuning based on video multimodal interface fine-tuning data to obtain an interface fine-tuning result, wherein the video multimodal interface fine-tuning data includes the visual input and the question-answering data collected from the large language model; as well as Instruction fine-tuning is performed based on video multimodal instruction fine-tuning data, wherein the video multimodal interface instruction data includes the visual input and question-answering data with instructions collected from the large language model.
15. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the chatting method according to any one of claims 12 to 14 are implemented.
16. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the chatting method described in any one of claims 12 to 14 are implemented.
17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the chatting method according to any one of claims 12 to 14 are implemented.
Citation Information
Patent Citations
Video understanding text generation method based on hierarchical representation network
CN113779310A
Visual question and answer training method and system and computer readable storage medium
CN114840651A
Video analysis method and device based on time boundary perception and large language model
CN117636217A
Spatial-Temporal Reasoning Through Pretrained Language Models for Video-Grounded Dialogues
US20210174162A1
Unified Vision and Dialogue Transformer with BERT
US20210232773A1
Cited By
Multi-modal large model processing method and device, storage medium and program product
CN119314117A
Multi-modal data storage control device and method of intelligent agent, intelligent agent and equipment
CN119356624A
Video analysis processing system and method based on multi-modal large model
CN119540831A
First-person video question answering method and system based on cross-view semantic alignment
CN119622024A
Efficient student Chinese painting appreciation system and appreciation method based on LLaVA model
CN119784230A
