A knowledge-enhanced video question answering method and system based on two-stage verification

Abstract

This invention discloses a knowledge-enhanced video question answering method and system based on two-stage verification, comprising: acquiring the video to be processed and the question, and extracting a set of keyframes from the video; performing image-text retrieval based on the keyframe set and an external text knowledge base; constructing independent inputs for each retrieval text, and generating candidate entities, inference statements, and candidate answers in parallel using a lightweight visual language model; performing a first-stage verification on the candidate answers using a heavyweight visual language model, calculating a reliability score, and filtering to obtain a set of highly reliable candidates; performing a second-stage verification using cross-modal similarity between candidate entities and keyframes, and calculating an entity alignment score; and selecting the target candidate answer as the final output result based on the entity alignment score. This invention reduces the latency of video retrieval enhancement generation and inference while improving the consistency between candidate answers and real entities in the video, effectively suppressing fine-grained entity confusion errors, and is suitable for knowledge-intensive video question answering scenarios.

Description

Technical Field

[0001] This invention relates to the field of visual language modeling technology, and in particular to a knowledge-enhanced video question-answering method and system based on two-stage verification. Background Technology

[0002] With the development of Vision-Language Models (VLMs), these models are now capable of generating natural language responses in image understanding, video understanding, and visual reasoning tasks. However, when dealing with external knowledge such as encyclopedic knowledge, specialized knowledge, entity attributes, and historical information, relying solely on the knowledge stored within the model's parameters often fails to yield accurate answers. Therefore, researchers typically introduce Retrieval-Augmented Generation (RAG) mechanisms. These mechanisms retrieve text related to the visual content from external knowledge bases, and then the Vision-Language Model combines the visual input with the retrieved text to generate the answer.

[0003] Existing multimodal RAG solutions typically include the following process: First, visual features are extracted from video or image content; second, several candidate documents are retrieved from an external document library based on the visual features or question text; third, the retrieved documents are concatenated with the original visual input and the question to form a long contextual input; finally, a single high-parameter visual language model completes cross-modal understanding, knowledge integration, and answer generation in one go.

[0004] Another type of existing approach attempts to use speculative decoding, where a smaller model generates a draft, which is then validated by a larger model. However, in multimodal RAG scenarios, existing methods typically focus more on improving generation speed and do not adequately consider the fine-grained alignment between visual entities and retrieved text entities.

[0005] To address the lack of external knowledge in knowledge-intensive video question answering, existing technologies typically employ a Retrieval-Augmented Generation (RAG) framework. This type of method usually first retrieves relevant textual knowledge from the question or multimodal input, then inputs the video, question, and all retrieved text into a large-scale visual language model to generate the final answer.

[0006] Existing standard video RAG schemes typically include the following steps: First, extract several representative frames from the input video, or encode the entire video directly; second, retrieve relevant documents from an external knowledge base based on the question text, video content, or a joint representation of both; then, concatenate the video, question, and multiple retrieved documents directly into a unified input; finally, use a single large visual language model to jointly model the above long context and generate the final answer.

[0007] In addition, some existing technologies attempt to introduce speculative decoding into multimodal question answering scenarios. This type of approach typically involves a lightweight model generating one or more candidate answers first, followed by a heavyweight model validating, rescoring, or reordering the candidate answers to reduce the overhead of a large model directly generating the complete answer.

[0008] In addition, in the image and text retrieval part, existing technologies often use cross-modal embedding models such as Contrastive Language-Image Pretraining (CLIP) to map video frames and text into a unified vector space and perform recall based on cosine similarity.

[0009] Existing knowledge-enhanced video question answering technologies typically employ the following approach: first, keyframes are extracted from the video; then, similarity searches are performed between these keyframes and an external knowledge base to obtain relevant text; finally, the video, question, and retrieved text are directly concatenated and input into a visual language model to generate the answer. While this type of technology can compensate for the lack of internal knowledge within the model to some extent, it still has the following drawbacks in practical applications:

[0010] (1) Low inference efficiency and long response latency. Existing technologies typically concatenate multiple retrieved external texts with video content and question text before inputting them into a large-parameter visual language model for one-time inference. Since the length of the concatenated input context increases significantly, and the visual language model has high computational complexity and inference overhead for long context inputs, it directly leads to a significant increase in model inference latency and increased consumption of computational resources, resulting in low overall question-answering efficiency and difficulty in meeting the application requirements of efficient video question answering scenarios.

[0011] (2) Fine-grained mismatches can easily occur between the retrieved text and the real entities in the video, leading to a decrease in the accuracy of the answers. Existing technologies usually rely on the overall similarity between visual content and text content for retrieval. However, this retrieval method can only guarantee that the text is roughly related to the video in terms of topic, and cannot guarantee that the entities described in the retrieved text are completely consistent with the real entities in the video. Since there are often fine-grained objects in the video that are similar in appearance, similar in category, or semantically related, when the retrieval results contain text that is similar to the real object but is not the same entity, the model is easily interfered with by incorrect knowledge in the subsequent reasoning process, which leads to entity confusion, attribute errors, or factual bias in the answers, reducing the accuracy of the question-and-answer results.

[0012] (3) The lack of an effective verification mechanism for the reliability of candidate answers makes it easy to output seemingly reasonable but actually incorrect answers. Most existing standard retrieval enhancement generation methods directly generate a single answer after obtaining the retrieval text, lacking a dedicated step for further reliability judgment of the generated results. Since the model generation process can be affected by erroneous retrieval text, noise information, or similar entity information, even if the final answer seems fluent in language and complete in logic, it may be based on erroneous evidence or erroneous entities. Due to the lack of a verification mechanism for the reliability of candidate answer reasoning, existing technologies struggle to identify and eliminate such "seemingly correct but actually wrong" answers in a timely manner.

[0013] (4) The lack of a secondary verification mechanism for the consistency between visual entities and text entities makes it difficult to suppress entity substitution errors. Existing technologies typically do not perform specific consistency verification between the text entities upon which the answer is based and the visual entities in the video when generating answers. Therefore, when there are incorrect entities in the retrieved text that are similar to the video object, the model easily substitutes the incorrect entity for the real entity in the video to complete the answer. Due to the lack of secondary verification at the entity level, the system struggles to filter out answers inconsistent with the video content from multiple candidate answers, thus affecting the authenticity and stability of the final output.

[0014] (5) Existing technologies struggle to simultaneously achieve both high accuracy in answering questions and efficient reasoning. Without introducing external knowledge, visual language models struggle to answer questions that rely on encyclopedic knowledge, background knowledge, or fine-grained factual knowledge, resulting in low accuracy. While directly concatenating multiple external texts using existing standard retrieval enhancement methods can improve the accuracy of some questions, it also leads to a significant increase in reasoning latency and computational costs. Therefore, it is evident that existing technologies in knowledge-enhanced video question answering scenarios still struggle to achieve both high accuracy and low reasoning overhead, and the overall technical performance remains significantly inadequate.

[0015] In summary, existing standard video RAG methods typically concatenate multiple retrieval texts with the video and question before inputting them into a large visual language model for unified generation. As the number of retrieval texts increases, the model input length increases significantly, leading to higher inference latency, increased GPU memory overhead, and higher deployment costs. Furthermore, in fine-grained entity scenarios such as animal species, aircraft models, landmarks, and industrial parts, the retrieval module easily recalls textual knowledge that is similar to but not consistent with the real entities in the video, resulting in entity confusion in the model's generated answers. While some existing speculative decoding or candidate re-ranking methods can reduce generation costs to some extent, they usually only score or filter candidate answers at the linguistic level, lacking explicit modeling of the consistency between video entities and candidate answer entities, thus limiting their error correction capabilities in multimodal entity confusion scenarios.

[0016] Therefore, the present invention aims to propose a technical solution that can balance reasoning efficiency and answer accuracy, reduce the latency of video knowledge question answering, improve the consistency between candidate answers and real entities in the video, and enhance the system's ability to suppress fine-grained entity errors.

[0017] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0018] To address the shortcomings of existing technologies, the present invention aims to provide a knowledge-enhanced video question answering method based on two-stage verification, thereby solving problems such as high latency in long contextual reasoning, entity confusion caused by retrieval noise, lack of fine-grained entity consistency verification, and insufficient reliability of candidate answers in existing video retrieval enhancement generation schemes.

[0019] To achieve the above objectives, the present invention adopts the following technical solution:

[0020] This invention provides a knowledge-enhanced video question-answering method based on two-stage verification, comprising the following steps:

[0021] S1: Obtain the video to be processed and the problem The video is then used to extract keyframes and output a set of keyframes. ;

[0022] S2: Set keyframes With external text knowledge base The image-text search is performed as input, and the search result is a collection of search text. ;

[0023] S3: Question ,video and retrieve text sets For each search text, candidate entities, inference statements, and candidate answers are generated as input, and a set of candidate answers is output. ;

[0024] S4: Using the candidate answer set ,video and the problem Using this as input, perform the first-stage validation, output the reliability score for each candidate answer, and filter to obtain a set of highly reliable candidates. ;

[0025] S5: Using a highly reliable candidate set and keyframe set Using this as input, perform the second-stage validation and output the entity alignment score corresponding to each highly reliable candidate answer;

[0026] S6: Using a highly reliable candidate set Align the entity score with the input to determine the final answer. And output;

[0027] The above steps are executed sequentially in chronological order. In particular, S3 processes different retrieved texts in parallel to reduce the overall inference latency.

[0028] Furthermore, in step S1, the video preprocessing module receives the original video. As input, output keyframe set This module first reads the video frame by frame, representing the video as a frame sequence. Subsequently, for the current frame And the previous frame Calculate similarity; when the similarity is below a preset threshold... At that time, the current frame Add as a keyframe to the keyframe set ;

[0029] Inter-frame similarity is represented by color histogram similarity or grayscale histogram correlation, and the keyframe extraction rules are as follows:

[0030] ;

[0031] in, This represents the similarity calculation function. This represents the keyframe determination threshold; the input for this step is the video. The output is a set of keyframes. .

[0032] Furthermore, in step S2, the image and text retrieval module receives a set of keyframes. and external text knowledge base As input, the output is a collection of search texts. This module first encodes keyframe images using a cross-modal embedding model to obtain keyframe vectors. Simultaneously, it encodes document titles, document summaries, document text, or combinations thereof from the knowledge base to obtain text vectors. Then, it calculates the similarity between each keyframe vector and the text vectors in the knowledge base, and selects the first... One candidate text;

[0033] The cross-modal embedding model is a contrastive language-image pre-trained model, and the image-text similarity calculation formula is as follows:

[0034] ;

[0035] in, Indicates a keyframe. Represents text, Cosine similarity is used to represent the search results. After single-keyframe retrieval, the candidate texts returned from multiple keyframes are aggregated, deduplicated, and sorted to obtain the final retrieval text set. The input for this step is a set of keyframes. and knowledge base The output is a collection of search texts. .

[0036] Furthermore, in step S3, the drafting module receives video. ,question and retrieve text sets As input, output a set of candidate answers. ; For the search text set Each search text in Each input is constructed independently, and candidate generation is performed in parallel by a lightweight visual language model, specifically as follows:

[0037] For each search text The draft generation module first calculates the content based on the video. and search text Extract candidate entities ; then according to the video ,question Candidate entities and search text Generate inference statements Finally, based on the video... ,question Candidate entities and reasoning statements Generate candidate answers The correspondence is as follows:

[0038] ;

[0039] ;

[0040] ;

[0041] After performing the above process on all searched texts, a set of candidate answers is obtained:

[0042] ;

[0043] The input for this step is a video. ,question and retrieve text sets The output is a set of candidate answers. Each candidate answer contains entities and intermediate inference results for further use by the subsequent verification module.

[0044] Furthermore, in step S4, the reliability verification module receives the set of candidate answers. ,video and the problem As input, output the reliability score of each candidate answer and the set of highly reliable candidates. This module uses a weighted visual language model to validate each candidate answer, rather than regenerating the complete answer.

[0045] Specifically, for the set of candidate answers any candidate , will candidate answers Candidate entities Reasoning Statement Along with the video and the problem Input both values ​​into the weighted visual language model and calculate the probability of the first generated label for the model's output "yes" and "no", denoted as . and ; candidate answers The reliability score is defined as follows:

[0046] ;

[0047] After obtaining the reliability scores of all candidate answers, further adjustments are made based on the preset tolerance boundary parameters. Constructing a highly reliable candidate set The filtering rules are as follows:

[0048] ;

[0049] in, This is used to control the range of candidates entering the second stage of verification; the input to this step is the set of candidate answers. ,video and the problem The output is a list of reliability scores and a set of highly reliable candidates. .

[0050] Furthermore, in step S5, the entity alignment module receives a set of highly reliable candidates. and keyframe set As input, the module outputs the entity alignment score corresponding to each highly reliable candidate answer; this module is used to determine whether the candidate entities involved in the candidate answers are consistent with the real visual entities in the video.

[0051] Specifically, for a highly reliable candidate set any candidate answer Extract its corresponding candidate entities , will candidate entities Encode into entity vectors; simultaneously, the keyframe set Each keyframe in the algorithm is encoded as a visual vector; then, the cross-modal similarity between the candidate entity vector and the visual vector of each keyframe is calculated, and the maximum value is taken as the candidate answer. The entity alignment score is calculated using the following formula:

[0052] ;

[0053] The input for this step is a highly reliable candidate set. and keyframe set The output is a list of entity alignment scores.

[0054] Furthermore, in step S6, the decision output module receives the highly reliable candidate set. Align the entity score with the entity as input and output the final answer. Specifically, the entity alignment scores of each candidate answer in the highly reliable candidate set are compared, and the candidate answer with the highest entity alignment score is selected as the final answer; the selection rules are as follows:

[0055] ;

[0056] When multiple candidate answers have the same or similar entity alignment scores, a secondary ranking is performed with reference to the reliability scores obtained in the first stage of verification to improve the stability of the final output; the input for this step is a set of highly reliable candidates. Align the scores with the entities and output the final answer. .

[0057] A knowledge-enhanced video question answering system based on two-stage verification is disclosed. The system includes: a video preprocessing module, a text-image retrieval module, a draft generation module, a reliability verification module, an entity alignment module, and a decision output module. Each module is deployed on the same server or in a distributed computing environment. The lightweight visual language model and the heavyweight visual language model are models with different parameter scales within the same model family, or combinations of heterogeneous models from different model families. The cross-modal embedding model adopts CLIP, SigLIP, SigLIP2, or other models with unified text-image representation capabilities.

[0058] By adopting the above technical solution, the present invention has the following beneficial effects:

[0059] (1) This invention reduces inference costs by splitting multiple search texts into multiple independent inputs and generating candidate answers in parallel by a lightweight visual language model, thus avoiding the high latency and high memory overhead caused by long context splicing of multiple documents in standard video RAG.

[0060] (2) By introducing a two-stage verification mechanism of "reliability verification + entity alignment verification", this invention ensures that candidate answers not only need to be consistent at the language and reasoning level, but also need to be consistent with the video visual content at the entity level, thereby effectively improving the accuracy and stability of the final answer.

[0061] (3) By explicitly generating candidate entities and reasoning statements in the drafting stage, this invention provides verifiable intermediate evidence for the subsequent verification process, thereby improving the interpretability of the system and the effectiveness of candidate reordering.

[0062] (4) The present invention can suppress two types of fine-grained errors: cross-entity knowledge transfer and entity substitution. It is especially suitable for knowledge-intensive video question-answering scenarios where entities are easily confused, such as animal recognition, vehicle recognition, landmark recognition, and industrial object recognition.

[0063] (5) The retrieval module, drafting module, verification module and entity alignment module in this invention can be deployed in a decoupled manner, which can adapt to visual language models with different parameter scales and different types of cross-modal embedding models, and has good engineering scalability. Attached Figure Description

[0064] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0065] Figure 1 A flowchart of a knowledge-enhanced video question-answering method based on two-stage verification provided in an embodiment of the present invention;

[0066] Figure 2 This is a framework diagram of a knowledge-enhanced video question-answering system based on two-stage verification, provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Before providing a detailed explanation of this application, the following is a clarification of the technical terms used in this application:

[0069] (1) Visual Language Model (VLM): refers to a multimodal model that can jointly process image, video and text input and output text results.

[0070] (2) Retrieval-Augmented Generation (RAG): refers to a technical framework that first retrieves relevant knowledge from an external knowledge base and then uses the retrieval results to enhance the generation process.

[0071] (3) Speculative Decoding: refers to a decoding strategy in which a lightweight model first generates candidate results quickly, and then a heavyweight model verifies, corrects or reorders them to reduce the overall inference latency.

[0072] (4) Candidate entities: refers to the descriptions of objects, categories, species, models, landmarks or other core entities that are identified from the video and search text during the drafting and generation stage and are related to the candidate answers.

[0073] (5) Reasoning statement: refers to the intermediate reasoning process generated by the model around the candidate entities, questions and retrieval text, which is used to explain the basis for the formation of candidate answers.

[0074] (6) Reliability score: refers to the score calculated by the weighted visual language model based on candidate answers, candidate entities, reasoning statements, videos and questions, which reflects whether candidate answers are reasonable and credible.

[0075] (7) Entity alignment score: refers to the similarity score between the candidate entity and the video keyframe in the cross-modal vector space, which is used to characterize the degree of consistency between the candidate entity and the real visual object in the video.

[0076] (8) Cross-Entity Transfer Error: This refers to an error type in which the model recognizes the real entity in the video, but incorrectly uses external knowledge corresponding to other similar entities for reasoning.

[0077] (9) Entity Substitution Error: This refers to a type of error in which the model is affected by erroneous search text, directly replacing the real entity in the video with the erroneous entity, and generating the final answer based on the erroneous entity.

[0078] like Figure 1 and Figure 2 As shown, the system of the present invention generally includes an input unit, a video preprocessing module, an image and text retrieval module, a draft generation module, a reliability verification module, an entity alignment module, and a decision output module.

[0079] The input unit is used to receive the video to be processed. User issues and external text knowledge bases The video preprocessing module is connected to the input unit and is used to process video data. Extracting keyframe sets The image and text retrieval module is connected to the video preprocessing module and an external text knowledge base, and is used to perform retrieval based on keyframe sets. From the knowledge base Recall relevant texts to obtain a set of retrieved texts. The drafting module is connected to the image and text retrieval module and is used to process the retrieved text set. Each retrieved text generates candidate entities, inference statements, and candidate answers. The reliability verification module, connected to the drafting generation module, performs a first-stage verification of the candidate answer set and outputs a reliability score. The entity alignment module, connected to both the reliability verification module and the video preprocessing module, performs a second-stage verification of the candidate entities corresponding to highly reliable candidate answers and outputs an entity alignment score. The decision output module, connected to both the reliability verification module and the entity alignment module, determines the final answer based on the verification results. .

[0080] The system input is video. ,question and external text knowledge base The system output is the final answer. The internal data flow of the system consists of: a set of keyframes. , Retrieve text collection Candidate answer set Highly reliable candidate set And the final answer .

[0081] like Figure 1 As shown, the overall process of the method of the present invention includes the following steps:

[0082] S1: Obtain the video to be processed and the problem The video is then used to extract keyframes and output a set of keyframes. ;

[0083] S2: Set keyframes With external text knowledge base The image-text search is performed as input, and the search result is a collection of search text. ;

[0084] S3: Question ,video and retrieve text sets For each search text, candidate entities, inference statements, and candidate answers are generated as input, and a set of candidate answers is output. ;

[0085] S4: Using the candidate answer set ,video and the problem Using this as input, perform the first-stage validation, output the reliability score for each candidate answer, and filter to obtain a set of highly reliable candidates. ;

[0086] S5: Using a highly reliable candidate set and keyframe set Using this as input, perform the second-stage validation and output the entity alignment score corresponding to each highly reliable candidate answer;

[0087] S6: Using a highly reliable candidate set Align the entity score with the input to determine the final answer. And output it.

[0088] The above steps are executed sequentially in chronological order. Among them, S3 can be executed in parallel for processing different searched texts to reduce the overall inference latency.

[0089] In step S1, the video preprocessing module receives the raw video. As input, output keyframe set This module first reads the video frame by frame, representing the video as a frame sequence. Subsequently, for the current frame And the previous frame Calculate similarity; when the similarity is below a preset threshold... At that time, the current frame Add as a keyframe to the keyframe set .

[0090] In a preferred embodiment, inter-frame similarity is characterized by color histogram similarity or grayscale histogram correlation, and the keyframe extraction rules are as follows:

[0091]

[0092] in, This represents the similarity calculation function. This represents the keyframe determination threshold. Through the above processing, representative visual content can be extracted without retaining all video frames, reducing the computational load of subsequent retrieval and verification. The input for this step is the video. The output is a set of keyframes. .

[0093] In step S2, the image and text retrieval module receives the keyframe set. and external text knowledge base As input, the output is a collection of search texts. This module first encodes keyframe images using a cross-modal embedding model to obtain keyframe vectors; simultaneously, it encodes document titles, document summaries, document text, or combinations thereof from the knowledge base to obtain text vectors. Then, it calculates the similarity between each keyframe vector and the text vectors in the knowledge base, and selects the first... 10 candidate texts.

[0094] In a preferred embodiment, the cross-modal embedding model is a Contrastive Language-Image Pretraining (CLIP) model, and the image-text similarity calculation formula is as follows:

[0095]

[0096] in, Indicates a keyframe. Represents text, This represents cosine similarity. After completing single-keyframe retrieval, the candidate texts returned from multiple keyframes are aggregated, deduplicated, and sorted to obtain the final retrieved text set. The input for this step is a set of keyframes. and knowledge base The output is a collection of search texts. .

[0097] In step S3, the drafting module receives the video. ,question and retrieve text sets As input, output a set of candidate answers. Unlike existing technologies that directly concatenate multiple search texts into a single long context to generate a unified answer, this invention targets search text sets. Each search text in Independent inputs are constructed separately, and candidate generation is performed in parallel by a lightweight visual language model.

[0098] Specifically, for each search text The draft generation module first calculates the content based on the video. and search text Extract candidate entities ; then according to the video ,question Candidate entities and search text Generate inference statements Finally, based on the video... ,question Candidate entities and reasoning statements Generate candidate answers The correspondence is as follows:

[0099]

[0100]

[0101]

[0102] After performing the above process on all searched texts, a set of candidate answers is obtained:

[0103]

[0104] The input for this step is a video. ,question and retrieve text sets The output is a set of candidate answers. Each candidate answer includes entities and intermediate inference results for further use by the subsequent verification module.

[0105] In step S4, the reliability verification module receives the candidate answer set. ,video and the problem As input, output the reliability score of each candidate answer and the set of highly reliable candidates. This module uses a weighted visual language model to validate each candidate answer, rather than regenerating the complete answer.

[0106] Specifically, for the set of candidate answers any candidate , will candidate answers Candidate entities Reasoning Statement Along with the video and the problem Input both values ​​into the weighted visual language model and calculate the probability of the first generated label for the model's output "yes" and "no", denoted as . and Candidate answers The reliability score is defined as follows:

[0107]

[0108] After obtaining the reliability scores of all candidate answers, further adjustments are made based on the preset tolerance boundary parameters. Constructing a highly reliable candidate set The filtering rules are as follows:

[0109]

[0110] in, This is used to control the range of candidates that proceed to the second stage of validation. The input to this step is the set of candidate answers. ,video and the problem The output is a list of reliability scores and a set of highly reliable candidates. .

[0111] In step S5, the entity alignment module receives the high-reliability candidate set. and keyframe set As input, the module outputs the entity alignment score for each highly reliable candidate answer. This module is used to determine whether the candidate entities involved in the candidate answers are consistent with the real visual entities in the video.

[0112] Specifically, for a highly reliable candidate set any candidate answer Extract its corresponding candidate entities , will candidate entities Encode into entity vectors; simultaneously, the keyframe set Each keyframe in the algorithm is encoded as a visual vector. Then, the cross-modal similarity between the candidate entity vector and the visual vector of each keyframe is calculated, and the maximum value is taken as the candidate answer. The entity alignment score is calculated using the following formula:

[0113]

[0114] The input for this step is a highly reliable candidate set. and keyframe set The output is a list of entity alignment scores.

[0115] In step S6, the decision output module receives the highly reliable candidate set. Align the entity score with the entity as input and output the final answer. Specifically, the entity alignment scores of each candidate answer in the highly reliable candidate set are compared, and the candidate answer with the highest entity alignment score is selected as the final answer. The selection rules are as follows:

[0116]

[0117] In a preferred embodiment, when multiple candidate answers have the same or similar entity alignment scores, a secondary ranking can be performed with reference to the reliability scores obtained in the first stage of verification to improve the stability of the final output. The input to this step is a set of highly reliable candidates. Align the scores with the entities and output the final answer. .

[0118] This invention establishes corresponding suppression principles for two types of fine-grained entity confusion errors in knowledge-intensive video question answering. The first type of error is cross-entity knowledge transfer error, where the model recognizes a real entity in the video but references external knowledge from erroneous search text during inference. This type of error typically performs poorly at the linguistic consistency level, and is therefore prioritized for suppression through the first-stage reliability verification. The second type of error is entity substitution error, where the model is completely misled by erroneous search text, causing the entities in the candidate answers to be inconsistent with the real entities in the video. This type of error may still be consistent at the linguistic level, and is therefore prioritized for suppression through the second-stage entity alignment verification. By mapping the two types of errors to a two-stage verification process, this invention can perform targeted filtering of errors from different sources.

[0119] Without departing from the basic concept of the present invention, the present invention also has several alternative implementation methods, which can also achieve the purpose of the present invention and should be regarded as falling within the protection scope of the present invention.

[0120] (1) In terms of key frame extraction, the key frame extraction method based on histogram threshold can be replaced by lens boundary detection, motion feature clustering, semantic key frame selection or representative frame extraction method based on temporal segmentation.

[0121] (2) In terms of image and text retrieval, the image and text similarity retrieval based on CLIP can be replaced by SigLIP, SigLIP2 or other unified image and text embedding models. Alternatively, the retrieval objects can be expanded from document titles to document paragraphs, structured attribute tables, knowledge graph triples or video clip description text.

[0122] (3) In terms of draft generation, lightweight visual language models and heavy visual language models can be either combinations of models with different parameter scales in the same family or heterogeneous combinations of different model families, as long as they can complete the candidate generation and candidate verification functions respectively.

[0123] (4) In terms of reliability verification, in addition to calculating the reliability score based on the probability of the first generated label “yes / no”, binary classification discriminant head, sequence-level consistency score, generative scorer or multidimensional consistency evaluation model can also be used to realize the candidate reliability judgment.

[0124] (5) In terms of entity alignment, in addition to using global cross-modal similarity between candidate entities and keyframes, fine-grained alignment methods such as region-level alignment, detection box-level alignment, segmentation region alignment, or introducing target detection results and visual grounding information can also be used.

[0125] (6) In terms of candidate ranking, in addition to adopting a two-stage sequential strategy of “first reliability screening and then entity alignment ranking”, the final answer decision can also be made by weighted fusion of reliability score and entity alignment score, cascade re-ranking or learnable sorter.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge-enhanced video question-answering method based on two-stage verification, characterized in that, Includes the following steps: S1: Obtain the video to be processed and the problem The video is then used to extract keyframes and output a set of keyframes. ; S2: Set keyframes With external text knowledge base The image-text search is performed as input, and the search result is a collection of search text. ; S3: Question ,video and retrieve text sets For each search text, candidate entities, inference statements, and candidate answers are generated as input, and a set of candidate answers is output. ; S4: Using the candidate answer set ,video and the problem Using this as input, perform the first-stage validation, output the reliability score for each candidate answer, and filter to obtain a set of highly reliable candidates. ; S5: Using a highly reliable candidate set and keyframe set Using this as input, perform the second-stage validation and output the entity alignment score corresponding to each highly reliable candidate answer; S6: Using a highly reliable candidate set Align the entity score with the input to determine the final answer. And output; The above steps are executed sequentially in chronological order. In particular, S3 processes different retrieved texts in parallel to reduce the overall inference latency.

2. The knowledge-enhanced video question-answering method based on two-stage verification according to claim 1, characterized in that, In step S1, the video preprocessing module receives the raw video. As input, output keyframe set This module first reads the video frame by frame, representing the video as a frame sequence. Subsequently, for the current frame And the previous frame Calculate similarity; when the similarity is below a preset threshold... At that time, the current frame Add as a keyframe to the keyframe set ; Inter-frame similarity is represented by color histogram similarity or grayscale histogram correlation, and the keyframe extraction rules are as follows: ； in, This represents the similarity calculation function. This represents the keyframe determination threshold; the input for this step is the video. The output is a set of keyframes. .

3. The knowledge-enhanced video question-answering method based on two-stage verification according to claim 1, characterized in that, In step S2, the image and text retrieval module receives the keyframe set. and external text knowledge base As input, the output is a collection of search texts. This module first encodes keyframe images using a cross-modal embedding model to obtain keyframe vectors. Simultaneously, it encodes document titles, document summaries, document text, or combinations thereof from the knowledge base to obtain text vectors. Then, it calculates the similarity between each keyframe vector and the text vectors in the knowledge base, and selects the first... One candidate text; The cross-modal embedding model is a contrastive language-image pre-trained model, and the image-text similarity calculation formula is as follows: ； in, Indicates a keyframe. Represents text, Cosine similarity is used to represent the search results. After single-keyframe retrieval, the candidate texts returned from multiple keyframes are aggregated, deduplicated, and sorted to obtain the final retrieval text set. The input for this step is a set of keyframes. and knowledge base The output is a collection of search texts. .

4. The knowledge-enhanced video question-answering method based on two-stage verification according to claim 1, characterized in that, In step S3, the drafting module receives the video. ,question and retrieve text sets As input, output a set of candidate answers. ; For the search text set Each search text in Each input is constructed independently, and candidate generation is performed in parallel by a lightweight visual language model, specifically as follows: For each search text The draft generation module first calculates the content based on the video. and search text Extract candidate entities ; then according to the video ,question Candidate entities and search text Generate inference statements Finally, based on the video... ,question Candidate entities and reasoning statements Generate candidate answers The correspondence is as follows: ； ； ； After performing the above process on all searched texts, a set of candidate answers is obtained: ； The input for this step is a video. ,question and retrieve text sets The output is a set of candidate answers. Each candidate answer contains entities and intermediate inference results for further use by the subsequent verification module.

5. The knowledge-enhanced video question-answering method based on two-stage verification according to claim 1, characterized in that, In step S4, the reliability verification module receives the candidate answer set. ,video and the problem As input, output the reliability score of each candidate answer and the set of highly reliable candidates. This module uses a weighted visual language model to validate each candidate answer, rather than regenerating the complete answer. Specifically, for the set of candidate answers any candidate , will candidate answers Candidate entities Reasoning Statement Along with the video and the problem Input the weighted visual language model together, calculate the probability of the first generated label for the model outputting "yes" and "no", and denot them as follows: and ; candidate answers The reliability score is defined as follows: ； After obtaining the reliability scores of all candidate answers, further adjustments are made based on the preset tolerance boundary parameters. Constructing a highly reliable candidate set The filtering rules are as follows: ； in, This is used to control the range of candidates entering the second stage of verification; the input to this step is the set of candidate answers. ,video and the problem The output is a list of reliability scores and a set of highly reliable candidates. .

6. The knowledge-enhanced video question-answering method based on two-stage verification according to claim 1, characterized in that, In step S5, the entity alignment module receives the high-reliability candidate set. and keyframe set As input, the module outputs the entity alignment score corresponding to each highly reliable candidate answer; this module is used to determine whether the candidate entities involved in the candidate answers are consistent with the real visual entities in the video. Specifically, for a highly reliable candidate set any candidate answer Extract its corresponding candidate entities , will candidate entities Encode into entity vectors; simultaneously, the keyframe set Each keyframe in the algorithm is encoded as a visual vector; then, the cross-modal similarity between the candidate entity vector and the visual vector of each keyframe is calculated, and the maximum value is taken as the candidate answer. The entity alignment score is calculated using the following formula: ； The input for this step is a highly reliable candidate set. and keyframe set The output is a list of entity alignment scores.

7. The knowledge-enhanced video question-answering method based on two-stage verification according to claim 1, characterized in that, In step S6, the decision output module receives the highly reliable candidate set. Align the entity score with the entity as input and output the final answer. ; Specifically, the entity alignment scores of each candidate answer in the highly reliable candidate set are compared, and the candidate answer with the highest entity alignment score is selected as the final answer; the selection rules are as follows: ； When multiple candidate answers have the same or similar entity alignment scores, a secondary ranking is performed with reference to the reliability scores obtained in the first stage of verification to improve the stability of the final output; the input for this step is a set of highly reliable candidates. Align the scores with the entities and output the final answer. .

8. A system applying the knowledge-enhanced video question-answering method based on two-stage verification as described in any one of claims 1-7, characterized in that, The system includes: a video preprocessing module, an image and text retrieval module, a draft generation module, a reliability verification module, an entity alignment module, and a decision output module; each module is deployed on the same server or in a distributed computing environment; the lightweight visual language model and the heavyweight visual language model are models with different parameter scales in the same model family, or are heterogeneous model combinations in different model families; the cross-modal embedding model adopts CLIP, SigLIP, SigLIP2, or other models with unified image and text representation capabilities.

Images