Interactive question answering system based on video understanding and working method thereof

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to an interactive question answering system based on video understanding and a working method thereof. The front-end interaction layer is used for interaction between the system and a user; the processing layer is used for realizing interactive questions and answers based on video understanding through a large model; the content in the knowledge enhancement layer is stored in a text form to form a knowledge base, secondary retrieval is carried out according to the question of the user and a video retrieval result, the retrieval result, the question of the user and a video retrieval fragment are input into a multi-modal large model together, and a final reply is generated; the method aims at solving the problems that free questions of users cannot be dynamically responded, semantic information of video pictures and audios is ignored, and the ability of understanding video time sequence contexts is lacked.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to an interactive question-answering system based on video understanding and a working method thereof. Background Art

[0002] With the explosive growth of online video content, the interactive models and traditional question-and-answer systems of existing video platforms are no longer able to meet users' increasingly complex information needs. Current mainstream video platforms still rely primarily on one-way content playback. To gain a deeper understanding of a specific segment, users must repeatedly fast-forward and rewind, tediously navigating the timeline, or exit the video and rely on external tools like search engines for assistance. This fragmented interaction significantly reduces information acquisition efficiency.

[0003] Traditional question-and-answer systems based on subtitle retrieval have significant limitations. First, they can only answer questions at pre-set time points, making them difficult to adapt to dynamic and ever-changing user needs. When users ask more personalized and flexible questions, the system often fails to respond accurately, resulting in a stilted interactive experience. Second, these systems rely solely on subtitle text for analysis and answering, completely ignoring the rich semantic information in the video, such as composition, color and lighting, ambient sound effects, and character intonation. For example, in film and television commentary scenes, key information such as character micro-expressions and the atmosphere of the scene cannot be captured by the system, resulting in a monotonous and lacking depth in the answers. Third, the system lacks the ability to understand the temporal context of the video, unable to analyze the video content as a whole. It struggles to grasp dynamic information such as plot development and evolving character relationships, resulting in answers that lack coherence and logic, failing to meet users' demand for a deep understanding of the video content. Therefore, innovative interaction models and intelligent question-and-answer technologies are urgently needed to enable more efficient and intelligent interaction between users and video content. Summary of the Invention

[0004] The present invention provides an interactive question-answering system based on video understanding and a working method thereof, which cannot dynamically respond to users' free questions, ignores the semantic information of video images and audio, and lacks the ability to understand the temporal context of the video.

[0005] The present invention is achieved through the following technical solutions: An interactive question-answering system based on video understanding, comprising: a front-end interaction layer, a processing layer, and a knowledge enhancement layer; The front-end interaction layer is used for interaction between the system and the user; The processing layer is used to implement interactive question-answering based on video understanding through a large model; The knowledge enhancement layer is used to generate a final response; The processing layer includes a context awareness module and a multimodal data encoding and query module; The knowledge enhancement layer includes a static knowledge base and a dynamic knowledge base, wherein the static knowledge base pre-stores domain structured data; The dynamic knowledge base crawls relevant encyclopedia data in real time.

[0006] Furthermore, the context-aware module uses a Transformer network to construct a video temporal memory pool, which stores multimodal features of the last N minutes.

[0007] Furthermore, the multimodal data encoding and query module converts text, voice and video into vector embedding representations. For the duration of all information blocks retrieved, all modalities will be processed separately, and the text and downsampled video will be input into the multimodal large model for analysis, and answers will be generated based on the questions and knowledge base.

[0008] Furthermore, the text embedding model and the multimodal embedding model are used for text data. Specifically, the text is converted into a high-dimensional vector and then encoded using the BERT, BGE-1.5 or CLIP model. Two encoding methods are used: one is text embedding encoding, and the other is multimodal embedding encoding through the CLIP model. A re-ranking model was introduced based on the rough ranking to further improve the accuracy of the retrieval results.

[0009] Furthermore, multimodal embedding coding is used for video data. Specifically, the video is temporally and spatially downsampled. Each second of video is saved as an image and images with too large a resolution are spatially downsampled. The image is encoded using CLIP, which converts visual features into vector representations in a high-dimensional semantic space: multimodal embedding coding.

[0010] Furthermore, multimodal embedding encoding is used for audio data. Specifically, the audio is encoded into a vector through AudioCLIP and queried through text content: that is, multimodal embedding encoding.

[0011] A method for operating an interactive question-answering system based on video understanding, the method comprising the following steps: Step 1: The user provides a timestamp for text or voice input at the front-end interaction layer; Step 2: The system inputs the user's interactive content into the context-aware module of the processing layer, calculates the dynamic event window and selects the video segment; Step 3: Based on the video segment selected in step 2, the multimodal data encoding and query module of the processing layer is started to extract text features, speech features, and video features, and convert them into vector embedding representations and perform multimodal feature fusion; Step 4: The static knowledge base and dynamic knowledge base of the knowledge enhancement layer are searched again based on the user's questions and the results of the video search; Step 5: Combine the multimodal features from step 3 with the results of the secondary retrieval in step 4 to perform hybrid knowledge reasoning; Step 6: Based on the hybrid knowledge reasoning in step 5, the response generator is obtained and the final reply is generated.

[0012] Furthermore, for video analysis, step 5 is specifically as follows: Step 5.1: If the multimodal data encoding and query module processes the video, it will simultaneously perform video analysis and knowledge enhancement layer retrieval based on the input question; Step 5.2: Perform evidence-weighted fusion of the video analysis results and the retrieval of the knowledge enhancement layer; Step 5.3: Generate candidate answers based on the weighted fusion results of evidence in step 5.2; Step 5.4: Evaluate the credibility of the candidate answers generated in step 5.3; Step 5.5: If the confidence level is greater than the set threshold, output the final answer; Step 5.6: If the confidence level is less than the set threshold, the output is "unanswerable" or manual labeling is requested.

[0013] Furthermore, the video analysis in step 5.1 is specifically as follows: The dynamic event window is calculated according to the input question, the video clip of the window is extracted, and the video clip is multimodally encoded.

[0014] Furthermore, the knowledge enhancement layer retrieval in step 5.1 is specifically to call the knowledge graph query of the knowledge enhancement layer according to the input question, perform entity recognition based on the query result, and implement relationship path search based on the recognition result.

[0015] The beneficial effects of the present invention are: The present invention can dynamically understand the context: it automatically adjusts the analysis time window according to the user's question (such as extending it forward by 10 seconds or 1 minute), and synchronously analyzes the video image, audio and subtitles through a multimodal model.

[0016] The present invention can realize multimodal collaborative analysis and establish a vision-speech-text trimodal feature fusion model. The video uses a visual feature model to extract key frame spatiotemporal features, extract audio features, identify speech emotions and key words, and combine text features with ASR subtitles and semantic coding.

[0017] The present invention can generate adaptive responses: the large language model generates natural language answers based on video semantic understanding and supports voice broadcast output. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the method of the present invention.

[0019] Figure 2 It is the hybrid knowledge reasoning flow chart of the present invention.

[0020] Figure 3 It is a logical diagram of the multimodal feature fusion of the present invention. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details.

[0022] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0023] It should also be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] The following is attached to this application specification Figure 1-3 , clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of this application.

[0025] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below. Implementation Method 1

[0026] This embodiment provides an interactive question-answering system based on video understanding, the system comprising: a front-end interaction layer, a processing layer, and a knowledge enhancement layer; The front-end interaction layer is used for interaction between the system and the user; The processing layer is used to implement interactive question-answering based on video understanding through a large model; The knowledge enhancement layer stores the content in text form to form a knowledge base. A secondary search is performed based on the user's question and the video search results. The search results, along with the user's question and the video search clip, are input into the multimodal large model to generate the final response. The processing layer includes a context awareness module and a multimodal data encoding and query module; The large model of the processing layer includes a large language model, a multimodal model, a text embedding model, a multimodal embedding model and a re-ranking model; Large language models: Both input and output are natural language. Their role is to generate satisfactory answers based on user input. In practice, large language models rely on massive amounts of text data for deep pre-training, building an extremely complex and extensive neural network architecture and knowledge base.

[0027] Multimodal model: The input of the multimodal model not only covers natural language, but also includes various types of data such as images and audio. Its output form is also rich and diverse, and it can generate results that integrate multiple modal information.

[0028] Text embedding models convert text into numerical vectors, encoding the text information into a low-dimensional dense vector space. Their core function is to accurately capture the semantic features of the text, condensing the multi-level information contained in the text, including vocabulary, grammar, and semantics, into a vector representation.

[0029] Multimodal Embedding Model: This model breaks through the limitations of single-text processing and integrates multiple modalities, such as images, audio, and video, with text data. It first uses a convolutional neural network through a specialized perception module to extract visual features from images and a Fourier transform to convert audio into spectral features. The model integrates features from different modalities into the same vector space. This model can mine the inherent connections and complementary information of multimodal data, embedding the fusion of multimodal data into a low-dimensional vector space to generate a more comprehensive and accurate information representation, enabling users to conduct cross-modal searches. When users enter query information, whether it is a text description, an image, or an audio clip, the model can respond quickly.

[0030] Reranking models receive the result list from the basic search model and, through carefully designed feature engineering, extract features from multiple dimensions, including text relevance, semantic match, and user behavior data. Using sophisticated machine learning or deep learning algorithms, they re-score and re-rank the initial results, prioritizing those that best meet user needs, are highly relevant, and offer high-quality results. This significantly improves the accuracy and effectiveness of information retrieval and recommendation systems, providing users with more valuable output.

[0031] The knowledge enhancement layer includes a static knowledge base and a dynamic knowledge base. The static knowledge base pre-stores domain structured data (such as teaching material knowledge points corresponding to educational videos, pre-prepared FAQs, etc.); The dynamic knowledge base crawls relevant encyclopedia data in real time (with an API speed limit of 500ms / time).

[0032] Furthermore, the context-aware module uses a Transformer network to construct a video temporal memory pool, which stores multimodal features of the last N minutes.

[0033] In order to more efficiently process the temporal information in video data and accurately capture long-range dependencies, the Transformer network is innovatively used to construct a video temporal memory pool. The Transformer network has powerful parallel computing capabilities and excellent modeling capabilities for long-sequence data. It converts information of different modalities into vector representations and constructs a video temporal memory pool through a deep neural network based on the Transformer architecture. The attention mechanism in this network can construct association weights for vectors between different modalities, accurately capturing the dynamic relationship and temporal dependency of multimodal information in the video. During the training process, the model continuously optimizes the allocation of attention so that the features of different modalities complement and strengthen each other. Ultimately, the video temporal memory pool can efficiently integrate various types of information, not only accurately identify video content, but also deeply understand its semantic and emotional connotations, providing more powerful and intelligent support for applications such as video analysis and retrieval.

[0034] The video temporal memory pool can store multimodal features from the last N minutes. Multimodal features encompass multiple dimensions of information within a video, including but not limited to visual features such as the shape, color, and texture of objects; audio features such as frequency, pitch, and volume; and even other forms of information such as text descriptions. Storing these rich multimodal features enables a more comprehensive understanding of video content.

[0035] By default, N is set to 5, meaning the memory pool focuses on storing the multimodal features of the last 5 minutes of video. In practice, the value of N can be adjusted flexibly based on specific needs and scenarios. As new video data is continuously input, the memory pool dynamically updates the stored feature information, always retaining the latest and most representative portions.

[0036] Through this design, the Video Temporal Memory Pool effectively captures long-range dependencies. In video analysis tasks, long-range dependencies often contain critical information, such as the development of the video plot and the evolution of character relationships. Traditional processing methods may face difficulties in capturing these long-range dependencies. However, the Transformer-based Video Temporal Memory Pool, with its unique architecture and algorithms, can deeply explore temporal patterns and associated information in video data, providing stronger support for subsequent video analysis, understanding, and application.

[0037] Furthermore, the multimodal data encoding and query module converts text, voice and video into vector embedding representations. For the duration of all retrieved information blocks (the information blocks are formed by combining video, text, voice and other content after time alignment), all modalities will be processed separately, and the text and downsampled video will be input into the multimodal large model for analysis, and answers will be generated based on the questions and knowledge base.

[0038] Furthermore, the use of text embedding models and multimodal embedding models for text data is specifically to convert text into high-dimensional vectors, which can capture the logic, semantics and other information in the text while reducing the amount of data represented, thereby avoiding the problem of inaccurate recall by methods such as direct keyword detection; an ideal embedding model will make texts with similar semantics close to each other in the vector space, thereby improving the recall accuracy; the present invention uses models such as BERT, BGE-1.5 or CLIP to encode text; text is the main basis for retrieval by the context-aware module, so two encoding methods are used, one is text embedding encoding, which is only effective for text and has a high text semantic recall accuracy. Commonly used methods are BERT, BGE-1.5, etc.; the other is to implement multimodal embedding encoding through the CLIP model. This encoding method supports cross-modal text queries, such as querying images through text, etc., which facilitates the context-aware module to understand the content of video images; A re-ranking model was introduced based on the rough ranking to further improve the accuracy of the search results. The model comprehensively considers multi-dimensional features such as semantic matching, contextual association, and user behavior preferences to make fine adjustments to the candidate results. Through the optimization of deep learning algorithms, the re-ranking model can more keenly capture the user's true intentions, ensuring that the final presented content not only meets the search requirements but is also highly relevant and practical.

[0039] Furthermore, multimodal embedding coding is used for video data. Specifically, video information has a large redundancy. Therefore, when encoding the video, the video is first temporally and spatially downsampled. Each second of video is saved as an image, and images with too large a resolution are spatially downsampled. Next, the image is encoded using CLIP, which converts visual features into vector representations in a high-dimensional semantic space: multimodal embedding coding. This process links image content with language descriptions, allowing the model to understand complex scenes and details in the image. The CLIP model can capture semantic information in images and implement retrieval through text.

[0040] Furthermore, multimodal embedding encoding is used for audio data. Specifically, the audio is encoded into vectors using AudioCLIP and then searched using text content: this is multimodal embedding encoding. Because AudioCLIP's text embedding is compatible with the original CLIP, for text, only one text embedding and one CLIP multimodal embedding need to be generated.

[0041] Example 1: Online Education Scenario • User asked at 15 minutes and 30 seconds: "What are the conditions under which this theorem applies?" • The system automatically analyses the video segment from 14:50 to 15:30 and generates voice answers including formula derivations based on the courseware knowledge base.

[0042] Example 2: Film and TV Narration Scene • User pauses to ask: "Where was this scene filmed?" • The system identifies landmark buildings in the image and generates answers after searching the library of film and television production footage.

[0043] Example 3: Historical Information Search • User paused and asked: "Which novel by Marquez was mentioned earlier in the video?" • The system searches and answers based on the video content: One Hundred Years of Solitude; • Users can then ask further questions about the life and deeds of Marquez, and the system will summarize the answers based on the online search content. Implementation Method 2

[0044] This embodiment provides a working method of an interactive question-answering system based on video understanding, which includes the following steps: Step 1: The user provides a timestamp for text or voice input at the front-end interaction layer; Step 2: The system inputs the user's interactive content into the context-aware module of the processing layer, calculates the dynamic event window and selects the video segment; Step 3: Based on the video segment selected in step 2, the multimodal data encoding and query module of the processing layer is started to extract text features, speech features, and video features, and convert them into vector embedding representations and perform multimodal feature fusion; Step 4: The static knowledge base and dynamic knowledge base of the knowledge enhancement layer are searched again based on the user's questions and the results of the video search; Step 5: Combine the multimodal features from step 3 with the results of the secondary retrieval in step 4 to perform hybrid knowledge reasoning; Step 6: Based on the hybrid knowledge reasoning in step 5, the response generator is obtained and the final reply is generated.

[0045] Furthermore, for video analysis, step 5 is specifically as follows: Step 5.1: If the multimodal data encoding and query module processes the video, it will simultaneously perform video analysis and knowledge enhancement layer retrieval based on the input question; Step 5.2: Perform evidence-weighted fusion of the video analysis results and the retrieval of the knowledge enhancement layer; Step 5.3: Generate candidate answers based on the weighted fusion results of evidence in step 5.2; Step 5.4: Evaluate the credibility of the candidate answers generated in step 5.3; Step 5.5: If the confidence level is greater than the set threshold, output the final answer; Step 5.6: If the confidence level is less than the set threshold, the output is "unanswerable" or manual labeling is requested.

[0046] Furthermore, the video analysis in step 5.1 is specifically as follows: The dynamic event window is calculated according to the input question, the video clip of the window is extracted, and the video clip is multimodally encoded.

[0047] Furthermore, the knowledge enhancement layer retrieval in step 5.1 is specifically to call the knowledge graph query of the knowledge enhancement layer according to the input question, perform entity recognition based on the query result, and implement relationship path search based on the recognition result.

[0048] Specifically, the workflow embodiment Step S1: The user asks at 8 minutes and 15 seconds: Why does this chemical reaction require heating? Step S2: System dynamic analysis time window (6:30-8:15), where: The visual module recognizes the changes in the thermometer readings of the experimental device. The audio module detected three repetitions of the keyword "activation energy". The text module extracts reaction equations from subtitles.

[0049] Step S3: Search the knowledge base for the entry "Effect of Temperature on Reaction Rate" in the physical chemistry textbook Step S4: Generate Response: Heating provides activation energy (mentioned at 6:45 in the video), which increases the effective collision frequency of the reactant molecules. For details, see the Arrhenius equation diagram on the right.

[0050] Response speed is increased by more than 40% (compared to traditional full-text search solutions).

[0051] Question answering accuracy increased by 35% (through multimodal feature fusion).

[0052] Supports dynamic parsing of unstructured questions (such as "What are the steps just demonstrated?").

Claims

1. An interactive question-answering system based on video understanding, characterized in that: The system includes: a front-end interaction layer, a processing layer and a knowledge enhancement layer; The front-end interaction layer is used for interaction between the system and the user; The processing layer is used to implement interactive question-answering based on video understanding through a large model; The knowledge enhancement layer is used to generate a final response; The processing layer includes a context awareness module and a multimodal data encoding and query module; The knowledge enhancement layer includes a static knowledge base and a dynamic knowledge base, wherein the static knowledge base pre-stores domain structured data; The dynamic knowledge base crawls relevant encyclopedia data in real time.

2. The system according to claim 1, characterized in that The context perception module uses a Transformer network to construct a video temporal memory pool, which stores multimodal features of the last N minutes.

3. The system according to claim 1, characterized in that The multimodal data encoding and query module converts text, voice and video into vector embedding representations. For the duration of all information blocks retrieved, all modalities will be processed separately. The text and downsampled video will be input into the multimodal large model for analysis, and answers will be generated based on the questions and knowledge base.

4. The system according to claim 3, characterized in that The text embedding model and multimodal embedding model are used for text data. Specifically, the text is converted into a high-dimensional vector and then encoded using the BERT, BGE-1.5 or CLIP model. Two encoding methods are used: one is text embedding encoding, and the other is multimodal embedding encoding through the CLIP model. A re-ranking model was introduced based on the rough ranking to further improve the accuracy of the retrieval results.

5. The system according to claim 3, characterized in that: The use of multimodal embedding coding for video data involves temporal and spatial downsampling of the video, saving each second of video as an image and spatially downsampling images with excessive resolution. The image is encoded using CLIP, which converts visual features into vector representations in a high-dimensional semantic space: multimodal embedding coding.

6. The system according to claim 3, characterized in that: Multimodal embedding encoding is used for audio data. Specifically, audio is encoded into vectors through AudioCLIP and queried through text content: that is, multimodal embedding encoding.

7. The working method of the interactive question-answering system based on video understanding according to any one of claims 1 to 6, characterized in that: The working method comprises the following steps: Step 1: The user provides a timestamp for text or voice input at the front-end interaction layer; Step 2: The system inputs the user's interactive content into the context-aware module of the processing layer, calculates the dynamic event window and selects the video segment; Step 3: Based on the video segment selected in step 2, the multimodal data encoding and query module of the processing layer is started to extract text features, speech features, and video features, and convert them into vector embedding representations and perform multimodal feature fusion; Step 4: The static knowledge base and dynamic knowledge base of the knowledge enhancement layer are searched again based on the user's questions and the results of the video search; Step 5: Combine the multimodal features from step 3 with the results of the secondary retrieval in step 4 to perform hybrid knowledge reasoning; Step 6: Adaptive response generation is obtained based on the hybrid knowledge reasoning in step 5, and the final reply is generated.

8. The method according to claim 7, characterized in that: For video analysis, step 5 is specifically as follows: Step 5.1: If the multimodal data encoding and query module processes the video, it will simultaneously perform video analysis and knowledge enhancement layer retrieval based on the input question; Step 5.2: Perform evidence-weighted fusion of the video analysis results and the retrieval of the knowledge enhancement layer; Step 5.3: Generate candidate answers based on the weighted fusion results of evidence in step 5.2; Step 5.4: Evaluate the credibility of the candidate answers generated in step 5.3; Step 5.5: If the confidence level is greater than the set threshold, output the final answer; Step 5.6: If the confidence level is less than the set threshold, the output is "unanswerable" or manual labeling is requested.

9. The method according to claim 8, characterized in that The video analysis in step 5.1 is specifically as follows: The dynamic event window is calculated according to the input question, the video clip of the window is extracted, and the video clip is multimodally encoded.

10. The method according to claim 8, characterized in that: The knowledge enhancement layer retrieval in step 5.1 is specifically to call the knowledge graph query of the knowledge enhancement layer according to the input question, perform entity recognition based on the query result, and implement relationship path search based on the recognition result.

Images