An open vocabulary video question answering method based on heterogeneous graphs and multi-modal prompts

Abstract

The application discloses an open vocabulary video question and answer method based on a heterogeneous graph and multi-modal prompts, comprising: obtaining an input video and a natural language question, extracting video frame features, target object features and question semantic features; combining an external knowledge graph to construct a heterogeneous graph containing visual nodes, question nodes and knowledge nodes; performing feature projection and Transform type information propagation based on multi-class meta paths to update node representations in the heterogeneous graph; decomposing a question subgraph and a visual subgraph from the updated heterogeneous graph to generate text prompts and visual prompts; inputting the multi-modal prompts, the video, the question and the candidate answer into a question and answer reasoning model together, and outputting answers through an answer sorting or generation mechanism. The application can utilize the complementary prompts of "text description + image example" to alleviate text ambiguity and improve the reasoning ability of rare answers and unseen answers in open vocabulary video question and answer.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of video understanding, visual language modeling, knowledge graph reasoning, open vocabulary recognition, and video question answering, and more specifically to an open vocabulary video question answering method based on heterogeneous graphs and multimodal prompts. Background Technology

[0002] Video question answering tasks aim to automatically generate answers based on input video content and natural language questions. As the openness and complexity of real-world video content continue to increase, models often need to encounter concepts, objects, behaviors, or answer categories not present in the training set during the testing phase. Therefore, open-vocabulary video question answering technology requires models not only to understand the spatiotemporal information of the video and the semantics of the question, but also to possess the ability to generalize reasoning about rare and unseen categories.

[0003] Existing video question answering methods can generally be categorized into models fine-tuned based on task data, models pre-trained using visual language, and models incorporating external knowledge. Methods fine-tuned based on task data are easily limited by the distribution of answers in the training set, and their performance degrades significantly when the test answer belongs to an unseen category. While methods pre-trained using visual language can provide some cross-modal generalization ability, they still struggle to handle complex common-sense relationships and fine-grained visual differences without external knowledge constraints that match the question and video content.

[0004] Existing open-vocabulary video question answering methods often introduce external knowledge through text, such as generating text prompts based on similar words or knowledge base entries. This type of method suffers from two main drawbacks: First, there is a lack of a unified alignment mechanism between external knowledge from different sources, video visual information, and question semantic information, easily leading to mismatches at the semantic, modal, and structural levels. Second, simple text prompts are easily affected by the ambiguity of natural language, making it difficult to provide fine-grained information such as visual appearance, shape, and gestures. For example, the same word may correspond to completely different visual concepts in different contexts, and textual explanations alone are insufficient to fully eliminate ambiguity. Summary of the Invention

[0005] The present invention provides an open vocabulary video question answering method based on heterogeneous graphs and multimodal prompts, which can discover rare or unseen answer-related relationships through structured knowledge and supplement the visual details lacking in text prompts with image example prompts, and can at least solve one of the above-mentioned technical problems.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: An open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts includes the following steps: S1. Obtain the input video sequence and the corresponding natural language question. Use a visual encoder, an object detector, and a text encoder to obtain the visual features of the video frame, the features of the target object, and the semantic features of the question and candidate answer, respectively. Map the above features to a unified semantic space. S2. Based on the visual features of video frames and the semantic features of questions, a heterogeneous graph is constructed by combining an external knowledge graph. The heterogeneous graph includes visual nodes, question nodes, and knowledge nodes, and also includes multiple types of edges such as visual to knowledge, question to knowledge, knowledge to knowledge, and question to question. S3. Based on the preset meta-path set, message propagation is performed on the heterogeneous graph. Feature projection and information propagation based on self-attention are used to update the features of various nodes to obtain video representation and problem representation that integrate external knowledge. S4. Generate multimodal cues based on the updated heterogeneous graph, wherein the multimodal cues include text cues generated from the problem subgraph and visual cues generated from the visual subgraph; S5. Input the video representation, question representation, candidate answer representation, and multimodal prompts into the question-answering reasoning model to obtain the answer prediction results for open-vocabulary video question answering.

[0007] Further, in S1, the input video sequence is represented as V={I1,I2,...,I...} N}, where I i Let represent the i-th video frame, i∈1,2,…,N, where N represents the total number of video frames. The visual encoder is used to extract global visual features for each frame, and the object detector is used to detect the set O of salient target objects in each frame. t ={o t1 ,o t2 ,...,o tM}, and obtain the corresponding target object features F t ={f t1 ,f t2 ,...,f tM}, where tj represents the j-th target in the t-th frame, j∈1,2,…,M, and M represents the total number of targets in the t-th frame.

[0008] Furthermore, in S1, the natural language problem is decomposed into multiple problem words or phrase nodes through semantic dependency analysis. These multiple problem words or phrase nodes are encoded by a pre-trained text encoder to obtain corresponding semantic embeddings. The category name of the target object and the name of the external knowledge graph node are also embedded into a unified semantic space through the same text encoder.

[0009] Furthermore, in S2, the heterogeneous graph is represented as G={N} v N q Nk ,P}, where N v N represents the set of visual nodes. q N represents the set of problem nodes. k Let P represent the set of knowledge nodes, and let P represent the set of meta-paths. Visual nodes are initialized by visual features of video frames or features of target objects, while question nodes and knowledge nodes are initialized by text semantic embedding.

[0010] Furthermore, in the heterogeneous graph, the edges from vision to knowledge and the edges from questions to knowledge are established by calculating the semantic similarity of nodes. When the semantic similarity is greater than the threshold α, the corresponding nodes are connected. The edges from knowledge to knowledge are established based on the entity relationships in the external knowledge graph, and the edges from questions to questions are established based on the semantic dependencies of questions or the full connectivity between question nodes.

[0011] Furthermore, in S3, the preset meta-path set includes at least the knowledge-to-question meta-path P. kq Knowledge to Visual Meta-Path P kv Problem-Knowledge-Visual Metapath P qkv Visual-Knowledge-Problem Metapath P vkq Visual-Knowledge-Visual Metapath P vkv And the problem-knowledge-problem metapath P qkq ,in: P kq and P kv Used to propagate external prior knowledge to problem nodes and visual nodes; P qkv and P vkq Used to establish cross-modal associations between questions and visual information, bridged by knowledge nodes; P vkv and P qkq Used to explore the correlations between internal components of the same modality based on external knowledge.

[0012] Furthermore, in S3, each meta-path contains at most β knowledge nodes between the starting node and the ending node, and the knowledge relationship edges in the meta-path come from an external knowledge graph, where β is a configurable hyperparameter used to limit the path length and computational complexity; For any meta-path, the node feature sequence is first projected through a multilayer perceptron, and then the self-attention between nodes is calculated through a Transformer-style information propagation module. The self-attention generates query vectors, key vectors, and value vectors based on the projected features, and obtains updated node features through multi-head attention, normalization layers, and feedforward layers.

[0013] Furthermore, in S4, the problem subgraph consists of the problem-knowledge-problem meta-path P.qkq And the knowledge-to-problem metapath P with semantic relevance to the problem node exceeding a threshold α kq The visual subgraph obtained by splicing together the knowledge-to-visual meta-path P is a visual subgraph. kv and the visual-knowledge-visual metapath P vkv It was pieced together.

[0014] Furthermore, in S4, the generation of the text prompt includes: organizing the paired nodes and edge relationships in the problem subgraph into a phrase set, inputting the phrase set into a pre-trained language model, and generating coherent text prompts to describe the semantics of the problem, the relationship between the scene and common sense. The generation of the visual cue includes: inputting the visual subgraph into a pre-trained language model to generate a visual scene description and related common sense description; calling a text-to-image generation model based on the visual scene description to generate at least one image example; then using a pre-trained visual encoder to extract the visual features of the image example; and obtaining a visual cue representation through an aggregation network, wherein: The text-to-image generation model includes a diffusion-based image generation model. The aggregation network includes a Transformer encoder and learnable classification labels. After generating K image examples for category c, the visual encoding features of the K image examples and the learnable classification labels are input into the Transformer encoder to obtain the visual cue vector for category c.

[0015] Furthermore, in step S5, the input template received by the question-answering reasoning model includes at least text prompts, visual prompt representations, video features, question text, and candidate answer text, and outputs the matching score of each candidate answer or directly outputs the natural language answer; The question-answering reasoning model employs a multi-candidate answer ranking loss during the training phase, ensuring that the similarity score of the correct answer is at least a preset interval higher than the similarity scores of other candidate answers. Furthermore, low-rank adaptation parameters are introduced only for a portion of the attention layers in the question-answering reasoning model during training, while the visual encoder and image generation model remain frozen.

[0016] The beneficial effects of this invention are reflected in: 1. By organizing videos, questions, and external knowledge in the same structure through heterogeneous graphs, the semantic and structural mismatch problems between multi-source information can be effectively alleviated.

[0017] 2. Through the multi-path propagation mechanism, external knowledge can be effectively transmitted along knowledge-to-problem, knowledge-to-visual, and cross-modal bridging paths, thereby enhancing information related to rare and unseen concepts.

[0018] 3. By combining textual and visual cues, the abstract semantics of the language description are preserved while fine-grained visual cues from the image examples are introduced, which can mitigate the impact of textual ambiguity.

[0019] 4. Without the need for additional manual annotation of unseen category samples, knowledge graphs and generative visual examples can be used to enhance the generalization ability of open vocabularies.

[0020] 5. During the training phase, only some parameters in the question-answering reasoning model need to be adapted, while the visual encoder and image generation model can remain frozen, making it easy to deploy in different video question-answering datasets and scenarios. Attached Figure Description

[0021] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0022] Figure 1 This is a schematic diagram of the overall process of the method according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The scope of protection of this invention is not limited to the specific network names, knowledge base names, parameter values, or dataset names in the following steps. Any implementation method that, under the concept of this invention, uniformly models video visual information, question text information, and external knowledge graphs as a heterogeneous graph, and generates text prompts and visual prompts based on the updated heterogeneous graph for open-vocabulary video question answering, should be understood as falling within the scope of protection of this invention.

[0025] It should be noted that the meaning of "and / or" throughout the text includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or a solution that simultaneously satisfies A and B. Furthermore, "multiple" refers to two or more. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0026] See Figure 1This invention provides an open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts, comprising the following steps: S1. Construction of video and question information.

[0027] This step is used to obtain basic features from the input video and the problem that can be used for subsequent heterogeneous graph construction. First, the input video frame sequence V={I1,I2,...,I...} is received. N} and the natural language problem X corresponding to the video. q The input video can come from short videos on the internet, instructional videos, surveillance videos, first-person perspective videos, or other video sources containing continuous visual events.

[0028] Before performing this step, the video can undergo preprocessing such as frame sampling, frame rate unification, size normalization, segment extraction, and feature caching. The above preprocessing does not change the core technical solution of this invention; its purpose is to improve the uniformity and computational efficiency of video inputs from different sources.

[0029] S11, Visual Feature Acquisition.

[0030] A visual encoder is used to encode each frame or sampled frame of the input video to obtain frame-level global visual features. The visual encoder can be a CLIP visual encoder, a video transformer, a convolutional neural network, a temporal convolutional network, or other models capable of outputting visual representations of the video.

[0031] Furthermore, an object detector is employed to identify salient objects in each frame. For frame t, I... t The detected set of objects can be represented as O t ={o t1 ,o t2 ,...,o tM}, whose corresponding visual features can be represented as F t ={f t1 ,f t2 ,...,f tM} where M represents the number of detected objects. The object detector can be GLIP, Faster R-CNN, DETR, Grounding-DINO, or other open-vocabulary object detection models.

[0032] S12, Problem semantic feature collection.

[0033] By performing word segmentation, part-of-speech analysis, dependency parsing, or semantic dependency analysis on natural language problems, the problem can be abstracted into a semantic structure composed of multiple problem nodes. Taking the problem "what is a woman cutting a vegetable into slices with" as an example, we can extract problem nodes such as "woman", "cutting", "vegetable", "slices", and "with", as well as semantic relationships such as actions, objects, and tools between these problem nodes.

[0034] A pre-trained text encoder is used to encode the question nodes, resulting in semantic embeddings. To facilitate cross-modal connections, the category names of the target objects and the names of external knowledge graph nodes are also embedded using the same text encoder, enabling similarity calculations for visual objects, question words, and knowledge entities within the same semantic space.

[0035] S2, Heterogeneous graph construction.

[0036] This step is used to organize video visual information, question semantic information, and prior knowledge from external knowledge graphs into a unified graph structure based on the heterogeneous graph construction module. The resulting heterogeneous graph can be represented as: G={N v N q N k ,P} Where, N v N represents the set of visual nodes. q N represents the set of problem nodes. k Let P represent the set of knowledge nodes and P represent the set of meta-paths. Visual nodes can be initialized by object visual features, frame-level visual features, or a fusion of both; question nodes are initialized by text embeddings of question words or phrases; knowledge nodes are initialized by entity embeddings from external knowledge graphs that are associated with questions, visual objects, or candidate answers.

[0037] The external knowledge graph is a commonsense knowledge graph, a concept knowledge graph, a domain knowledge base, or a combination thereof. In a preferred embodiment, the external knowledge graph uses ConceptNet. Related knowledge nodes in ConceptNet are queried based on the question word, the target object category name, and the candidate answer text, and several nodes with high semantic similarity are retained. To avoid interference from low-quality relationships, low-frequency relationships or relationships irrelevant to the task can be filtered out.

[0038] S21. Establishment of multiple edge types.

[0039] Edges in heterogeneous graphs include at least four types: vision-to-knowledge edges (v2k), question-to-knowledge edges (q2k), knowledge-to-knowledge edges (k2k), and question-to-question edges (q2q). Knowledge-to-knowledge edges are directly established based on relationships in the external knowledge graph, such as RelatedTo, IsA, UsedFor, or AtLocation relationships. Question-to-question edges are established based on semantic dependencies of questions, or fully connected edges can be established between all question nodes.

[0040] For both vision-to-knowledge and question-to-knowledge edges, the semantic similarity between visual nodes or question nodes and knowledge nodes is calculated. When the similarity is greater than a threshold α, a connection is established between the corresponding nodes. Semantic similarity can be achieved using cosine similarity, dot product similarity, bilinear matching, or other learnable matching functions. This connection method enables external knowledge related to the semantics of video objects and questions to enter the subsequent message propagation process.

[0041] S22, Heterogeneous graph initialization.

[0042] After establishing the nodes and edges, an initial heterogeneous graph is obtained. This initial heterogeneous graph simultaneously contains visual content, problem semantics, and external prior knowledge, with different types of nodes establishing explicit relationships through multi-type edges. Compared to external knowledge introduction methods that only generate textual descriptions, heterogeneous graphs can preserve the structural relationships between knowledge nodes and provide path constraints for cross-modal information propagation.

[0043] S3, Message propagation based on meta-path.

[0044] This step involves selecting appropriate information propagation paths on the constructed heterogeneous graph, enabling external knowledge to effectively update visual and problem nodes. Metapaths are defined as sequences of nodes of different types, indicating the interaction patterns between them.

[0045] S31, Construction of meta-path set.

[0046] In a preferred embodiment, the system sets up six types of metapaths: P kq P kv P qkv P vkq P vkv and P qkq Among them, P kq P represents the path from a knowledge node to a problem node. kv P represents the path from the knowledge node to the visual node. qkv P represents the path from the problem node through the knowledge node to the visual node. vkq P represents the path from the visual node through the knowledge node to the problem node. vkv P represents the internal path of a visual node connected by a knowledge node.qkq This represents the internal path of the question node connected by the knowledge node.

[0047] Through P kq and P kv It can directly propagate external knowledge to problem nodes and visual nodes; through P qkv and P vkq It can establish cross-modal connections between problems and vision, bridged by knowledge nodes; through P vkv and P qkq It can explore common-sense correlations between multiple components within the same modality. The number of knowledge nodes contained in each path can be controlled by the hyperparameter β to balance information coverage and computational complexity.

[0048] S32, Feature projection.

[0049] Since the initial features of different types of nodes may come from different encoders or different modalities, feature projection is first performed on the node features along the metapath. For example, its node feature sequence is denoted as , where the path origin This represents the i-th visual node. Indicates and Adjacent knowledge nodes, path endpoint This represents the j-th problem node. Indicates and For adjacent knowledge nodes, the projection process can be represented as:

[0050] Here, MLP stands for Multilayer Perceptron Module, which may include linear layers, normalization layers, non-linear activation functions, and Dropout layers. Through feature projection, the features of various nodes on the meta-path are mapped to a semantic space of a unified dimension, facilitating subsequent attention propagation.

[0051] S33, Transformer-style information dissemination.

[0052] A Transformer-style information propagation module is used to further model the relationships between nodes within a meta-path. Let the set of all meta-paths be P = {p1, p2, ..., p...} L}, where L represents the total number of meta-paths, and the projection feature sequence corresponding to the i-th meta-path is} Then the feature sequence is mapped to a query vector. Key vector Sum value vector :

[0053] in, , and These are the learnable parameters; Subsequently, the relationship weights between nodes within the meta-path are calculated using a self-attention mechanism, and the propagation representation is obtained:

[0054] in, This is the latent space representation obtained after computation, where d represents the dimension of the key vector. It is a normalized exponential function.

[0055] Following the self-attention operation, a normalization layer, a feedforward layer, and another normalization layer can be sequentially connected, expanding into a multi-head attention structure. This allows the model to focus on knowledge relationships within the meta-path from different semantic perspectives. The propagation results are used to update the features of the corresponding visual nodes, question nodes, and knowledge nodes.

[0056] S4. Multimodal prompt generation.

[0057] This step involves generating text and visual cues based on the updated heterogeneous graph using a multimodal cue generation module. This enhances the question-answering reasoning model's ability to understand unseen entities and relationships. Unlike traditional methods that only provide text cues, the multimodal cues in this invention simultaneously incorporate abstract common sense at the linguistic level and concrete visual examples at the image level.

[0058] S41. Generation of problem subgraphs and visual subgraphs.

[0059] After message propagation, the semantic relevance between visual nodes, question nodes, and knowledge nodes is recalculated, and the relevant meta-paths are updated according to a threshold α. Subsequently, the question subgraph G is decomposed from the updated heterogeneous graph. q and visual subgraph G v .

[0060] When constructing the problem subgraph, retain the problem-knowledge-problem meta-path P. qkq It also identifies knowledge nodes whose semantic relevance to the question node is greater than a threshold α, and links the corresponding knowledge to the question meta-path P. kq With all P qkq By splicing, we obtain the problem subgraph G. q When constructing a visual subgraph, knowledge is transferred to the visual meta-path P. kv With the visual-knowledge-visual metapath P vkv By splicing, we obtain the visual subgraph G. v .

[0061] S42. Text prompt generation.

[0062] Subgraph G of the problem qThe paired nodes and edge relationships in the language are organized into phrase sets, which are then input into a pre-trained language model to generate coherent scene descriptions, problem semantic descriptions, or common-sense relationship descriptions. For example, for phrases like "person-cut-vegetable" and "cut-related to-knife," the language model can generate text prompts such as "person cutvegetable, cut is related to knife." These text prompts provide the question-answering reasoning model with abstract conceptual relationships and common-sense knowledge.

[0063] S43, Visual cues are generated.

[0064] visual subgraph G v A pre-trained language model is input to generate a visual scene description and related common-sense descriptions. Subsequently, a text-to-image generation model is used to generate several image examples based on these descriptions. The text-to-image generation model can be Stable Diffusion, a diffusion-based image generation model, a generative adversarial network, or another image generation model.

[0065] The generated image examples are then encoded using a pre-trained visual encoder, and visual cue representations are obtained through an aggregation network. For category c, if K image examples are generated... The visual cue vector can then be represented as:

[0066] in, For visual cue vectors, Represents a visual encoder. This indicates learnable classification tags. Index for image instances.

[0067] By aggregating multiple example images, visual cues can provide more specific cues in terms of color, shape, posture, scene, and motion appearance than text cues.

[0068] S5. Answer Reasoning and Training.

[0069] This step combines the generated multimodal prompts with the original video question-answering input to obtain open-vocabulary answer prediction results. In one implementation, text prompts, visual prompt representations, video global features, question text, and candidate answer text are concatenated as input to the question-answering inference model. The input template can be represented as: "Refer to the Text prompt and Visual Prompt to answer the question, Text prompt, Visual Prompt, Video features, Question text, candidate answer text".

[0070] The question-answering reasoning model can employ DeBERTa, BERT, T5, Vicuna, a multimodal large language model, or other language / visual language models. During training, a multi-candidate answer ranking loss is used, ensuring that the similarity score of the correct answer is at least an interval m higher than the similarity scores of other candidate answers. Optionally, a low-rank adaptation parameter LoRA is introduced only in the attention layer of the question-answering reasoning model, while keeping the visual encoder and image generation model frozen to reduce training costs and avoid destroying pre-trained knowledge.

[0071]

[0072] in, For sorting loss, This represents the matching score for the correct answer. represents the matching score of the negative candidate answer, and m represents the interval hyperparameter.

[0073] During the reasoning phase, the candidate answer with the highest score can be selected as the output, or the generative question answering model can directly output the natural language answer.

[0074] To further verify the feasibility and superiority of this method, the present invention is analyzed and illustrated with the following practical cases.

[0075] This embodiment uses four open video question-answering datasets—MSVD-QA, ActivityNet-QA, TGIF-QA, and MSRVTT-QA—as examples to illustrate the implementation process and effects of the method of this invention. CLIP is used as the video and text feature extractor, an object detector is used to extract salient objects from video frames, ConceptNet is used as the external commonsense knowledge graph, and a pre-trained language model is used to generate text prompts and visual scene descriptions.

[0076] In the specific parameter settings, the number of image examples K can be set to 4, the semantic similarity threshold α can be set to 0.8, and the maximum number of knowledge nodes β in the meta-path can be set to 3. ConceptNet retains several knowledge nodes with the highest semantic similarity to the question node or visual node, and filters out low-frequency or weakly related relationships. The text-to-image generation model can generate 512×512 example images, and the visual encoder filters image examples with high semantic matching.

[0077] Table 1 below shows the performance of the proposed method in open-vocabulary video question answering on different datasets in one embodiment. It can be seen that the proposed method not only achieves good overall accuracy but also maintains high recognition ability for Rare and Unseen answers. On the MSVD-QA dataset, the proposed method achieves a mAcc of 24.7%, with an accuracy of 25.4% for the Rare category and 18.4% for the Unseen category. On the TGIF-QA dataset, the proposed method further achieves a mAcc of 31.9%, with accuracies of 31.0% for the Rare category and 24.0% for the Unseen category. Compared to open-vocabulary video question answering methods that only focus on common categories, the proposed method maintains a high level of performance on unknown categories, indicating that the proposed heterogeneous graph modeling and multimodal prompting mechanism can effectively utilize cross-modal semantic associations, enhance the model's understanding of rare and unseen categories, and thus improve the generalization performance in open-vocabulary video question answering tasks.

[0078] Table 1. Performance results of the method of the present invention on different datasets for open-vocabulary video question answering.

[0079] Table 2 below shows the ablation results in one embodiment. Under the MSVD-QA-Unseen setting, the baseline model achieved only 16.1% accuracy. Introducing multimodal hints improved this to 16.9%, and further adding a knowledge heterogeneous graph improved it to 17.2%. The complete method reached 18.4%, an improvement of 2.3 percentage points compared to the baseline. On the overall MSVD-QA dataset metric mAcc, the complete method achieved 24.7%, an improvement of 3.1 percentage points compared to the baseline model's 21.6%.

[0080] Similarly, on the ActivityNet-QA dataset, the complete method achieves an accuracy of 8.3% in the Unseen category, a 2.7 percentage point improvement over the baseline model's 5.6%; the overall mAcc increases from 11.9% to 14.6%. These results demonstrate that introducing multimodal cues alone can provide richer visual semantic information, while knowledge heterogeneous graphs can further uncover potential relationships between different concepts. Combining the two can effectively enhance the model's understanding of unknown categories and complex video semantics, improving the robustness and generalization ability of open-vocabulary video question answering tasks.

[0081] Table 2. Ablation Experiment Results of the Core Module in the Method of the Invention

[0082] This method is applicable to open-vocabulary video question answering, zero-shot video question answering, few-shot video question answering, open-vocabulary video comprehension, video-based commonsense reasoning, and human-computer interaction scenarios that require identifying rare or unseen answer categories.

[0083] In practical applications, this method can be deployed in educational video question-answering systems, security video question-answering systems, robot vision question-answering systems, industrial inspection question-answering systems, and open-domain question-answering systems for internet videos. When users ask questions about rare objects, unseen actions, or complex common-sense relationships in a video, the system can utilize external knowledge graphs to discover relevant concepts and assist the reasoning model in outputting more accurate answers through text prompts and visual example hints.

[0084] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the open vocabulary video question-answering method based on heterogeneous graphs and multimodal prompts described above.

[0085] This invention also provides a computer device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the open vocabulary video question answering method based on heterogeneous graphs and multimodal prompts described above.

[0086] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of the open vocabulary video question answering method based on heterogeneous graphs and multimodal prompts described above.

[0087] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above-described open vocabulary video question answering method based on heterogeneous graphs and multimodal prompts.

[0088] It should be noted that those skilled in the art will understand that all or part of the steps implemented in the embodiments of the present invention can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented in hardware, it can be implemented entirely or partially by purchasing standard parts or modifications. When implemented in software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0089] In summary, this invention addresses the problems in existing open-vocabulary video question answering methods, such as difficulty in aligning multi-source knowledge, ambiguous text prompts, and a lack of fine-grained visual cues, leading to insufficient generalization ability for rare answers and answers not seen during the training phase. It proposes a multimodal prompt generation and answer reasoning technique that unifies the modeling of video, questions, and external knowledge. This method revolves around five stages: input construction, knowledge alignment, graph structure propagation, prompt generation, and answer reasoning. It achieves unified modeling of video, questions, and external knowledge at the heterogeneous graph level and complementary enhancement of text semantics and visual examples at the prompt level. This complementary prompting of "text description + image example" can alleviate text ambiguity and improve the reasoning ability for rare and unseen answers in open-vocabulary video question answering.

[0090] It should be understood that the examples and embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art can make various modifications or changes based on them. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. An open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts, characterized in that, Includes the following steps: S1. Obtain the input video sequence and the corresponding natural language question. Use a visual encoder, an object detector, and a text encoder to obtain the visual features of the video frame, the features of the target object, and the semantic features of the question and candidate answer, respectively. Map the above features to a unified semantic space. S2. Based on the visual features of video frames and the semantic features of questions, a heterogeneous graph is constructed by combining an external knowledge graph. The heterogeneous graph includes visual nodes, question nodes, and knowledge nodes, and also includes multiple types of edges such as visual to knowledge, question to knowledge, knowledge to knowledge, and question to question. S3. Based on the preset meta-path set, message propagation is performed on the heterogeneous graph. Feature projection and information propagation based on self-attention are used to update the features of various nodes to obtain video representation and problem representation that integrate external knowledge. S4. Generate multimodal cues based on the updated heterogeneous graph, wherein the multimodal cues include text cues generated from the problem subgraph and visual cues generated from the visual subgraph; S5. Input the video representation, question representation, candidate answer representation, and multimodal prompts into the question answering reasoning model to obtain the answer prediction results for open-vocabulary video question answering; In step S3, the preset meta-path set includes at least the knowledge-to-question meta-path P. kq Knowledge to Visual Meta-Path P kv Problem-Knowledge-Visual Metapath P qkv Visual-Knowledge-Problem Metapath P vkq Visual-Knowledge-Visual Metapath P vkv And the problem-knowledge-problem metapath P qkq ,in: P kq and P kv Used to propagate external prior knowledge to problem nodes and visual nodes; P qkv and P vkq Used to establish cross-modal associations between questions and visual information, bridged by knowledge nodes; P vkv and P qkq Used to explore the correlations between components within the same modality based on external knowledge; In S4, the problem subgraph consists of the problem-knowledge-problem meta-path P. qkq And the knowledge-to-problem metapath P with semantic relevance to the problem node exceeding a threshold α kq The visual subgraph obtained by splicing together the knowledge-to-visual meta-path P is a visual subgraph. kv and the visual-knowledge-visual metapath P vkv obtained by piecing together; In step S4, the generation of the text prompt includes: organizing the paired nodes and edge relationships in the problem subgraph into a phrase set, inputting the phrase set into a pre-trained language model, and generating a coherent text prompt to describe the semantics of the problem, the relationship between the scene and common sense. The generation of the visual cue includes: inputting the visual subgraph into a pre-trained language model to generate a visual scene description and related common sense description; calling a text-to-image generation model based on the visual scene description to generate at least one image example; then using a pre-trained visual encoder to extract the visual features of the image example; and obtaining a visual cue representation through an aggregation network, wherein: The text-to-image generation model includes a diffusion-based image generation model. The aggregation network includes a Transformer encoder and learnable classification labels. After generating K image examples for category c, the visual encoding features of the K image examples and the learnable classification labels are input into the Transformer encoder to obtain the visual cue vector for category c.

2. The open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts as described in claim 1, characterized in that, In S1, the input video sequence is represented as V={I1,I2,...,I...} N }, where I i Let represent the i-th video frame, i∈1,2,…,N, where N represents the total number of video frames. The visual encoder is used to extract global visual features for each frame, and the object detector is used to detect the set O of salient target objects in each frame. t ={o t1 ,o t2 ,...,o tM }, and obtain the corresponding target object features F t ={f t1 ,f t2 ,...,f tM }, where tj represents the j-th target in the t-th frame, j∈1,2,…,M, and M represents the total number of targets in the t-th frame.

3. The open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts as described in claim 2, characterized in that, In step S1, the natural language problem is decomposed into multiple problem words or phrase nodes through semantic dependency analysis. These multiple problem words or phrase nodes are encoded by a pre-trained text encoder to obtain corresponding semantic embeddings. The category name of the target object and the name of the external knowledge graph node are also embedded into a unified semantic space through the same text encoder.

4. The open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts as described in claim 1, characterized in that, In S2, the heterogeneous graph is represented as G={N} v N q N k ,P}, where N v N represents the set of visual nodes. q N represents the set of problem nodes. k Let P represent the set of knowledge nodes, and let P represent the set of meta-paths. Visual nodes are initialized by visual features of video frames or features of target objects, while question nodes and knowledge nodes are initialized by text semantic embedding.

5. The open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts as described in claim 4, characterized in that, In the heterogeneous graph, edges from vision to knowledge and edges from questions to knowledge are established by calculating the semantic similarity of nodes. When the semantic similarity is greater than the threshold α, the corresponding nodes are connected. Edges from knowledge to knowledge are established based on entity relationships in the external knowledge graph, and edges from questions to questions are established based on the semantic dependencies of questions or the full connectivity between question nodes.

6. The open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts as described in claim 1, characterized in that, In S3, each meta-path contains at most β knowledge nodes between the starting node and the ending node, and the knowledge relationship edges in the meta-path come from an external knowledge graph. Here, β is a configurable hyperparameter used to limit the path length and computational complexity. For any meta-path, the node feature sequence is first projected through a multilayer perceptron, and then the self-attention between nodes is calculated through a Transformer-style information propagation module. The self-attention generates query vectors, key vectors, and value vectors based on the projected features, and obtains updated node features through multi-head attention, normalization layers, and feedforward layers.

7. The open-vocabulary video question answering method based on heterogeneous graphs and multimodal prompts as described in claim 1, characterized in that, In step S5, the input template received by the question-answering reasoning model includes at least text prompts, visual prompts, video features, question text, and candidate answer text, and outputs the matching score of each candidate answer or directly outputs the natural language answer. The question-answering reasoning model employs a multi-candidate answer ranking loss during the training phase, ensuring that the similarity score of the correct answer is at least a preset interval higher than the similarity scores of other candidate answers. Furthermore, low-rank adaptation parameters are introduced only for a portion of the attention layers in the question-answering reasoning model during training, while the visual encoder and image generation model remain frozen.

Images