Video question and answer method for specific domain fine-grained heuristic prompt

Abstract

The invention discloses a video question answering method for specific domain fine-grained heuristic prompt. The method comprises the following steps: 1, designing a prompt template of a specific domain through data engineering; performing space-time cutting processing on the original video data; the space-time cutting treatment is carried out; 2, taking the prompt template and the clipped video data as input of an entity prompter and an action prompter to obtain a heuristic label; and 3, through a heuristic enhanced answer reasoning scheme, collaborative interaction between the visual language basic model and the entity prompter and between the visual language basic model and the action prompter is enhanced, that is, a label output by the visual language basic model is compared with a heuristic label, and the ability of the visual language basic model to accurately deduce answers is improved. According to the method, a large-scale model is converted into a dynamic knowledge engine, complex content can be efficiently and accurately analyzed, and attention to basic cross-modal elements is enhanced by integrating a fine-grained inspiration method in a specific field into design of a prompt template.

Description

Technical Field

[0001] The present invention relates to a deep neural network for video question answering, and more particularly to a video question answering method with fine-grained heuristic prompts for a specific domain. Background Art

[0002] The rapid development of large-scale text pre-training has significantly improved the capabilities of large language models, giving them unprecedented ability to understand and process complex language structures. Based on large language models, cross-modal pre-training models have also been significantly developed. For example, models such as CLIP and DALL-E have promoted a major leap in the field of deep learning by integrating visual and language data during the pre-training process. This progress has greatly promoted fields such as robotics, medical diagnosis, and interactive game environments that require deep cross-modal understanding.

[0003] Video question answering is a challenging and rapidly developing field in the integration of vision and language. It requires models to have both a deep understanding of a single modality and the ability to fully interact across modalities. It combines video understanding with language processing to infer reliable answers. Although traditional methods have made some progress in enhancing video language models and adaptive cross-modal interactions, the emergence of video language base models such as ALPRO and SiaSamRea marks a paradigm shift in models. These models are pre-trained on large-scale video-text datasets, endowing the models with massive cross-modal knowledge, setting a new standard for the interpretation of video content and significantly improving the performance of deep learning models in video question answering tasks. Despite the outstanding performance of video language base models, their general pre-training methods often fall short in certain specialized video question answering domains. This highlights the need to fully utilize these models through context-aware tuning. Integrated prompt engineering with visual language models offers the potential to address this problem by designing customized prompt templates to guide the model to tackle domain-specific challenges, thereby ensuring more accurate results. However, prompt-based methods rely heavily on the quality of prompt design. For tasks that require deep, context-specific insights, generic cues may not suffice, suggesting that more refined strategies are needed.

[0004] This paper proposes a novel framework that aims to enhance the basic video language model by domain-specific entity and action heuristics to meet the complex needs of video question answering. These needs stem from the challenges of handling diverse and complex video content. The model not only needs to understand time series and spatial relationships, but also needs to have the ability to handle nuanced and context-aware reasoning. This paper transforms large-scale models into dynamic knowledge engines that can efficiently and accurately analyze complex content, and enhances the focus on basic cross-modal elements by integrating domain-specific fine-grained heuristics into the design of prompt templates. Summary of the invention

[0005] The present invention provides a video question-answering method with fine-grained heuristic prompts for a specific domain. The architecture mainly includes: 1. Data engineering: optimizing the visual input of the heuristic knowledge engine through strategic video cropping and designing video entity action templates; 2. Heuristic knowledge engine: an entity action prompter, focusing on obtaining comprehensive heuristic information from data processed by data engineering; 3. Heuristic-enhanced answer reasoning: using the heuristic information obtained by the heuristic knowledge engine to achieve collaborative interaction between the video language basic model and the knowledge engine, thereby enhancing the answer reasoning performance of the overall model.

[0006] The technical solution adopted by the present invention to solve its technical problem is as follows.

[0007] Step (1), designing a prompt template for a specific domain through data engineering; performing spatiotemporal cropping on the original video data; the spatiotemporal cropping process: effectively segmenting the moments that may contain key action information related to answering questions, and effectively segmenting the areas of entity key information.

[0008] Step (2), using the prompt template and the cropped video data as inputs of the entity prompter and the action prompter, and obtaining the heuristic label, that is, capturing the temporal dynamics corresponding to the verbs and actions contained in the video through the action prompter, and identifying and tracking the spatial attributes corresponding to the nouns and entities in the video through the entity prompter.

[0009] The verbs and time dynamics are heuristic labels output by the action prompter.

[0010] The nouns and spatial attributes are heuristic labels output by the entity prompter.

[0011] Step (3) uses a heuristically enhanced answer reasoning scheme to strengthen the collaborative interaction between the visual language base model and the entity prompter and action prompter, that is, to compare the labels output by the visual language base model with the heuristic labels to improve the ability of the visual language base model to accurately infer the answer.

[0012] Furthermore, in step (1), a prompt template for a specific domain is designed through data engineering, and the original data is subjected to spatiotemporal cropping, as follows:

[0013] In order to effectively capture the key moments and regions that contain the key to accurately answer specific questions from the complete video while avoiding expensive computational requirements, the video is segmented in the temporal and spatial dimensions through specific cropping methods, and domain-specific entity hint templates and action hint templates are designed.

[0014] The specific cropping method: in order to maintain the temporal sensitivity of the model, the same spatial area between different key frames is continuously cropped to ensure that the changing actions can be captured in a concentrated manner; in order to fully obtain the information contained in the spatial dimension of the video, different areas in different frames of the video are cropped to provide spatial understanding resources related to entities in the video. The obtained video clips are represented as temporal cropped clips and spatial cropping fragments

[0015] The entity prompt template and action prompt template for designing a specific domain are as follows:

[0016] By designing predefined prompt templates, we assist the entity prompter and action prompter in identifying entities and actions contained in the video at the text level. The predefined prompt templates contain special placeholders that are used to replace common nouns and verbs in question-answer pairs in video question answering, ensuring a high degree of relevance between the prompt text and the question answering task, and enhancing the model's context-aware response capabilities.

[0017] Furthermore, step (2) is specifically as follows:

[0018] 2-1. The entity prompt template, action prompt template and cropped video serve the domain-specific inspiration generation stage, which relies on specific entity prompters EP and action prompters AP; the action prompter improves temporal accuracy by randomly cropping the same spatial regions on sparse video frames, while the entity prompter improves spatial sensitivity by extracting different regions from different frames. The entity prompter EP and action prompter AP share the same architecture and adopt a visual language base model with video branch and text branch.

[0019] For the video branch, both the entity prompter EP and the action prompter AP use 12-layer TimeSformer 224 To capture video features; TimeSformer 224 Take a video segment V∈R C×F×H×W As input, C represents the number of video frame channels, F represents the number of video frames, H and W represent the height and width of the video frame respectively; after random sampling, scaling, cropping and random masking, the computational efficiency and consistency are improved. The video embedding obtained by the action prompter AP is in, and Corresponding to the video global time embedding and time cropping numbers respectively, Indicates the time clipping number is The local temporal embedding of ; the video embedding obtained by the entity prompter EP is in, and is the video global spatial embedding and spatial cropping number, Indicates that the space clipping number is Local temporal embedding of

[0020] For the text branch, a multi-layer bidirectional Transformer is used to encode the text semantics. This encoding process generates corresponding text embeddings, including action prompt text embeddings and entity hint text embedding

[0021] The similarity function s(·) is optimized between the video embedding and the text embedding as follows:

[0022] s(v cls ,t cls ) = f v (v cls )·f t (t cls ) (Formula 1)

[0023] Among them, v cls represents global spatial embedding or global temporal embedding, t cls Represents action hint text embedding or entity hint text embedding;

[0024] Both AP and EP use the video-text pair dataset from the Internet and pre-train through the symmetric video-text contrast loss function. The symmetric contrast loss drives the encoder loss to converge in a low-dimensional space by bringing positive pairs (data on the diagonal) closer and separating negative pairs (data on the off-diagonal), thereby optimizing the model's multimodal alignment capabilities.

[0025] Symmetric Contrastive Loss for Visual-to-Text Modality The calculation is as follows:

[0026]

[0027] Symmetric Contrastive Loss for Text-to-Video Modality Calculated as:

[0028]

[0029] Where B represents the batch size, τ represents the learnable temperature parameter, represents the global spatial embedding or global temporal embedding of the i-th batch; represents the action prompt text embedding or entity prompt text embedding of the i-th batch; after being pre-trained on a large number of datasets, the prompter shows strong alignment ability for video and text data; in order to prevent external noise from affecting its performance, all parameters of the entity prompter EP and action prompter AP are frozen in the heuristic generation stage.

[0030] 2-2. The heuristic generation stage includes action heuristic generation and entity heuristic generation, which generate corresponding heuristic labels through the action prompter AP and the entity prompter EP respectively.

[0031] The action cueing extractor captures motion information by processing instantiated cue templates and processed video frames through a spatiotemporal cropping strategy. By computing the video temporal embedding and action prompt embed The maximum normalized similarity between is derived, and the formula is as follows:

[0032]

[0033] Where M represents the number of action cues, τ represents the learnable temperature parameter, and exp(.) represents the exponential function of e;

[0034] Entity-inspired labeling By computing the video spatial embedding and entity hint embedding The maximum normalized similarity between is obtained, and the formula is as follows:

[0035]

[0036] Where N represents the number of entities.

[0037] Furthermore, step (3) is as follows:

[0038] The heuristic-enhanced answer reasoning scheme uses the heuristic labels obtained in the heuristic generation phase to strengthen the collaborative interaction between the visual language base model and the entity prompter and action prompter; and improves the model's ability to accurately infer answers.

[0039] First, we use the video language base model to capture the contextual video embedding and text embedding, and then use the question-answering reasoner to interact and communicate the video and question-answering content. The question-answering reasoner fuses the video embedding and text embedding by stacking cross-modal alignment layers to explore informative cross-modal clues. The cross-modal alignment layer includes a self-attention layer, a cross-attention layer, and a feed-forward network to provide cross-modal fusion embedding Among them, e cls It is the link between the entity action prompter and the question answering reasoner. v represents the number of video embeddings, N t Represents the number of text embeddings.

[0040] The video language base model does not directly utilize the fusion embedding clsInstead of predicting the answer, the domain-specific entity action heuristic labels are used as the supervision targets of temporal action modeling TAM and spatial entity modeling SEM losses. TAM Loss enhances the model's ability to understand temporal action sequences, while SEM Loss improves the recognition of spatial relations between entities. These loss functions can transfer the latent knowledge in the prompter to the baseline model to guide the video language base model to perform more accurate reasoning in the video question answering task. Through two different classifiers c1(·) and c2(·), the normalized heuristic probability distributions of actions and entities are obtained:

[0041]

[0042] in, and They represent the action-inspired probability distribution and entity-inspired probability distribution obtained by the classifier respectively. m and e n corresponds to the mth action and nth entity in their respective distributions.

[0043] The TAM loss is calculated as the action-inspired probability distribution derived by the classifier Target distribution obtained with action prompter The cross entropy between:

[0044]

[0045] The SEM loss is calculated as the entity-inspired probability distribution obtained by the classifier And the target distribution obtained by the entity prompter The cross entropy between:

[0046]

[0047] Use a separate classifier that takes the heuristically enhanced fusion embedding cls As input, the formula is as follows:

[0048]

[0049] Will predict the answer Compared with the true label y, the prediction loss is calculated as follows:

[0050]

[0051] Combining the prediction loss with the inspiration loss, we get the complete training objective function of the model:

[0052]

[0053] Among them, α is a hyperparameter used to balance the weight between action and entity inspiration.

[0054] To explore a better trade-off, a dynamic gating mechanism is introduced that can dynamically adjust the influence of action and entity inspirations during the training phase according to the input data. This question-guided gating mechanism involves the following steps: Process the text input through a video language base model to extract text embeddings t cls , and passes this representation through a gating network consisting of two multilayer perceptrons, a dropout function, and an activation function:

[0055] g(t cls )=Sigmoid(MLP2(Dropout(MLP1(t cls )))) (Formula 12)

[0056] The gating mechanism can obtain a scalar from 0 to 1, representing the weight of the action prompt, and the weight of the entity prompt can be obtained by subtracting the weight of the action prompt from 1. Finally, these weights are applied to the TAM loss and SEM loss to obtain the final combined loss function formula:

[0057]

[0058] This adaptive gating scheme demonstrates its effectiveness in enhancing the adaptability and stability of the model by dynamically adjusting the influence of action and entity cues.

[0059] Beneficial effects of the present invention:

[0060] The present invention improves the model's ability to understand complex video content by introducing entity and action inspirations in specific fields, and enhances the model's accuracy and adaptability in video question-answering tasks by customizing prompt templates. It transforms a large-scale video language basic model into a dynamic knowledge engine that can flexibly respond to diverse video content changes. At the same time, it innovatively designs a fine-grained prompt method to ensure that the model can identify key entities and dynamic operations, improves the ability to solve cross-modal tasks, and especially makes the performance in the field of video question-answering more accurate and consistent. These effects have jointly promoted technological progress in the field of video question-answering and significantly improved the performance of the model in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the overall framework of the method of the present invention.

[0062] Figure 2 It is a schematic diagram of a physical action prompter in the method of the present invention.

[0063] Figure 3 It is a structural diagram of the heuristically enhanced answer reasoning in the method of the present invention. DETAILED DESCRIPTION

[0064] The detailed parameters of the present invention are further described in detail below.

[0065] like Figure 1 As shown, the present invention provides a video question answering method with fine-grained heuristic prompts for a specific domain.

[0066] Step (1), designing a prompt template for a specific domain through data engineering; performing spatiotemporal cropping processing on the original video data; the spatiotemporal cropping processing: effectively segmenting the moments that may contain key action information related to answering questions, and effectively segmenting the areas of key entity information;

[0067] 1-1. First, sample the original video frames. Each video is sampled into 16 frames. The obtained video segments are represented as time-cropped segments. and spatial cropping fragments

[0068] 1-2. Then we will design certain prompt templates to help the model identify entities and actions contained in the video from the text level. The entity and action prompt templates can be represented as "A video of [NONU]." and "A video of [VERB].". "[NONU]" and "[VERB]" represent special placeholders that can be systematically replaced with the top 1000 most common nouns and verbs extracted from the question-answer pairs in the video question-answering.

[0069] Step (2), using the prompt template and the cropped video data as inputs of the entity prompter and the action prompter, and obtaining the heuristic label, that is, capturing the temporal dynamics of the verbs and actions contained in the video through the action prompter, and identifying and tracking the spatial attributes of the nouns and entities in the video through the entity prompter; Figure 2 As shown, the details are as follows:

[0070] 2-1. For the video branch, both the action prompter and the entity prompter use 12-layer TimeSformer 224 To capture video features. It takes video clips As input, it is randomly sampled, scaled, cropped, and randomly masked to improve computational efficiency and consistency. The visual representation obtained by the action prompter is represented as in and They correspond to the global temporal embedding and temporal crop number of the video respectively. The visual embedding from the entity prompter is represented as in and is the video global spatial embedding and spatial crop number.

[0071] 2-2. For the text branch, a multi-layer bidirectional transformer is used to encode the text semantics. This process generates the corresponding prompt embeddings, and Corresponding to the ones used for action and entity cues. For cross-modal alignment, the similarity function s(·) is optimized between video and text embeddings as follows:

[0072] s(v cls ,t cls ) = f v (v cls )·f t (t cls )

[0073] Videos and prompts that match can receive higher similarity scores.

[0074] 2-3. Then we will enter the inspiration generation stage, which includes two sub-steps: action inspiration generation and entity inspiration generation, which involves generating corresponding heuristic labels through action and entity prompt modules respectively.

[0075] Action-inspired tags By computing the video temporal embedding and action prompt embed The maximum normalized similarity between is derived, and the formula is as follows:

[0076]

[0077] Entity heuristics by computing video spatial embeddings and all entity hints embedded The maximum normalized similarity between is obtained, and the formula is as follows:

[0078]

[0079] Step (3) uses a heuristically enhanced answer reasoning scheme to strengthen the collaborative interaction between the visual language base model and the entity prompter and action prompter, that is, to compare the labels output by the visual language base model with the heuristic labels to improve the ability of the visual language base model to accurately infer the answer, such as Figure 3 As shown, the details are as follows:

[0080] First, we use the video language base model to capture the contextual video embedding and text embedding, and then use the question-answering reasoner to interact and communicate the video and question-answering content. The question-answering reasoner fuses the video embedding and text embedding by stacking cross-modal alignment layers; the cross-modal alignment layer includes a self-attention layer, a cross-attention layer, and a feed-forward network to provide cross-modal fusion embedding Among them, e clsIt is the link between the entity action prompter and the question answering reasoner. v represents the number of video embeddings, N t represents the number of text embeddings;

[0081] The video language base model does not directly utilize the fusion embedding cls Instead of predicting the answer, the domain-specific entity-action heuristic labels are used as the supervision targets for the temporal action modeling TAM and spatial entity modeling SEM losses. Through two different classifiers c1(·) and c2(·), the normalized heuristic probability distributions of actions and entities are obtained:

[0082]

[0083] in, and They represent the action-inspired probability distribution and entity-inspired probability distribution obtained by the classifier respectively. m and e n corresponds to the mth action and nth entity in their respective distributions.

[0084] The TAM loss is calculated as the action-inspired probability distribution derived by the classifier Target distribution obtained with action prompter The cross entropy between:

[0085]

[0086] The SEM loss is calculated as the entity-inspired probability distribution obtained by the classifier And the target distribution obtained by the entity prompter The cross entropy between:

[0087]

[0088] Use a separate classifier that takes the heuristically enhanced fusion embedding cls As input, the formula is as follows:

[0089]

[0090] Among them, c3 represents the classifier, which will predict the answer Compared with the true label y, the prediction loss is calculated as follows:

[0091]

[0092] Combining the prediction loss with the inspiration loss, we get the complete training objective function of the model:

[0093]

[0094] Among them, α is a hyperparameter used to balance the weight between action and entity inspiration. It is generally set to 0.5, which means that the action and entity have the same weight.

[0095] This method also introduces a gating mechanism to process the text input through a video language base model to extract the global t cls Representation, and passing the representation through a multilayer perceptron, random dropout function, etc.:

[0096] g(t cls )=Sigmoid(MLP2(Dropout(MLP1(t cls ))))

[0097] The gating mechanism can obtain a scalar from 0 to 1, representing the weight of the action prompt, and the weight of the entity prompt can be obtained by subtracting the weight of the action prompt from 1. Finally, by applying these weights to the TAM loss and SEM loss, the final combined loss function formula can be obtained:

[0098]

[0099] Experimental data

[0100] Table 1 shows the comparative experimental results of the proposed method on the validation set and test set of the NEXT-QA dataset. ACC@C, ACC@T, and ACC@D represent the accuracy of causal questions, temporal questions, and descriptive questions, respectively, expressed in percentage (%). The best results are highlighted in bold, and the suboptimal results are underlined.

[0101] Table 1:

[0102]

[0103] Model performance comparison experiments on the NExT-QA dataset: Table 1 shows the comparison results of various models on the NExT-QA dataset, including non-pretrained models and pretrained visual language models. The experimental results show that our method always presents the best or near-best results in terms of overall performance indicators (Acc@All) and specific question types (causal, temporal, and descriptive). Our method shows a 1.0% improvement on the validation set and a 1.6% enhancement on the test set compared to the baseline model ALPRO. This performance improvement is mainly driven by additional action and entity heuristic information, which significantly enhances the fine-grained visual perception ability of the model. The experimental results highlight the unique advantages of our method in dealing with temporal questions, which stems from the ability of the action entity prompter to capture basic action information, which is crucial for answering temporal questions. It is worth noting that although these action information does not appear explicitly in the question or answer, it provides precise prompts that can significantly improve the response accuracy of the model. Similarly, our method also performs well on descriptive questions, thanks to the ability of the entity prompter, as it can identify a wide range of entity categories from visual data. By establishing relevant connections between entity heuristics and question-answer pairs, entity prompters can guide baseline models to solve entity-related questions, thereby enhancing the model's ability to solve description-based problems. Although our method may not outperform some methods in causal reasoning tasks, most of those leading methods have incorporated complex manually customized methods, including complex graph structure networks designed specifically for logical and causal reasoning. These methods require combining multiple numbers of input frames and have an inherent advantage in causal reasoning tasks. Although we did not win first place in this field, our method has also made significant progress, especially in solving complex causal reasoning that requires comprehensive analysis of various video elements.

Claims

1. A video question answering method with domain-specific fine-grained heuristics, characterized in that: The steps include: Step (1), designing a prompt template for a specific domain through data engineering; performing spatiotemporal cropping processing on the original video data; the spatiotemporal cropping processing: effectively segmenting the moments that may contain key action information related to answering questions, and effectively segmenting the areas of key entity information; Step (2), using the prompt template and the cropped video data as inputs of the entity prompter and the action prompter to obtain the heuristic label; Step (3) uses a heuristically enhanced answer reasoning scheme to strengthen the collaborative interaction between the visual language base model and the entity prompter and action prompter, that is, to compare the labels output by the visual language base model with the heuristic labels to improve the ability of the visual language base model to accurately infer the answer.

2. A video question answering method with domain-specific fine-grained heuristic prompts according to claim 1, characterized in that Step (1) is as follows: Through a specific cropping method, the video is segmented in time and space dimensions, and entity prompt templates and action prompt templates for specific domains are designed; The specific cropping method: by continuously cropping the same spatial area between different key frames, it ensures that the changing action can be captured in a concentrated manner; in order to fully obtain the information contained in the spatial dimension of the video, different areas in different frames of the video are cropped and the obtained video segments are represented as time cropped segments. and spatial cropping fragments The entity prompt template and action prompt template for designing a specific domain are as follows: By designing predefined prompt templates, the entity prompter and action prompter are assisted in identifying entities and actions contained in the video respectively at the text level; the predefined prompt templates contain special placeholders, which are used to replace common nouns and verbs in question-answer pairs in video question and answering.

3. A video question answering method with domain-specific fine-grained heuristic prompts according to claim 2, characterized in that Step (2) is as follows: The heuristic labels are the temporal dynamics of verbs and actions captured by the action prompter. The entity prompter is the spatial attribute corresponding to the noun and entity identified and tracked by the entity prompter.

4. The video question answering method with domain-specific fine-grained heuristic prompts according to claim 2 is characterized in that Step (2) is as follows: Entity hint templates, action hint templates, and cropped videos serve the domain-specific inspiration generation stage, which relies on specific entity hinters EP and action hinters AP; the action hinter improves temporal accuracy by randomly cropping the same spatial regions on sparse video frames, while the entity hinter improves spatial sensitivity by extracting different regions from different frames. The entity hinter EP and action hinter AP share the same architecture, using a visual language base model with video branches and text branches; For the video branch, both the entity prompter EP and the action prompter AP use 12-layer TimeSformer 224 To capture video features; TimeSformer 224 Take the video segment V∈RC ×F×H×W As input, C represents the number of video frame channels, F represents the number of video frames, H and W represent the height and width of the video frame respectively; after random sampling, scaling, cropping and random masking, the computational efficiency and consistency are improved; the video embedding obtained by the action prompter AP is in, and Corresponding to the video global time embedding and time cropping numbers respectively, Indicates that the time clipping number is The local temporal embedding of ; the video embedding obtained by the entity prompter EP is in, and is the video global spatial embedding and spatial cropping number, Indicates that the space clipping number is Local temporal embedding of For the text branch, a multi-layer bidirectional Transformer is used to encode the text semantics; this encoding process generates corresponding text embeddings, including action prompt text embeddings and entity hint text embedding The similarity function s(·) is optimized between the video embedding and the text embedding as follows: s(v cls ,t cls ) = f v (v cls )·f t (t cls ) (Formula 1) Among them, v cls represents global spatial embedding or global temporal embedding, t cls Represents action hint text embedding or entity hint text embedding; Symmetric Contrastive Loss for Visual-to-Text Modality The calculation is as follows: Symmetric Contrastive Loss for Text-to-Video Modality Calculated as: Where B represents the batch size, τ represents the learnable temperature parameter, represents the global spatial embedding or global temporal embedding of the i-th batch; represents the action prompt text embedding or entity prompt text embedding of the i-th batch; after being pre-trained on a large number of datasets, the prompter shows strong alignment ability for video and text data; in order to prevent external noise from affecting its performance, all parameters of the entity prompter EP and action prompter AP are frozen in the heuristic generation stage.

5. The video question answering method with domain-specific fine-grained heuristic prompts according to claim 4 is characterized in that: The heuristic generation stage includes action heuristic generation and entity heuristic generation, which generate corresponding heuristic labels through action prompter AP and entity prompter EP respectively; The action cueing unit captures the action information by processing the instantiated cue templates and the processed video frames through the spatiotemporal cropping strategy; Action-inspired tags By computing the video temporal embedding and action prompt embed The maximum normalized similarity between is derived, and the formula is as follows: Where M represents the number of action cues, τ represents the learnable temperature parameter, and exp(.) represents the exponential function of e; Entity-inspired labeling By computing the video spatial embedding and entity hint embedding The maximum normalized similarity between is obtained, and the formula is as follows: Where N represents the number of entities.

6. A video question answering method with domain-specific fine-grained heuristics according to claim 4 or 5, characterized in that Step (3) is as follows: The heuristic-enhanced answer reasoning scheme uses the heuristic labels obtained in the heuristic generation phase to strengthen the collaborative interaction between the visual language base model and the entity prompter and action prompter; First, we use the video language base model to capture the contextual video embedding and text embedding, and then use the question-answering reasoner to interact and communicate between the video and question-answering content. The question-answering reasoner fuses the video embedding and text embedding by stacking cross-modal alignment layers. The cross-modal alignment layer includes self-attention layer, cross-attention layer and feed-forward network to provide cross-modal fusion embedding Among them, e cls It is the link between the entity action prompter and the question answering reasoner. v represents the number of video embeddings, N t represents the number of text embeddings; The video language base model does not directly utilize the fusion embedding cls Instead of predicting the answer, the domain-specific entity-action heuristic labels are used as the supervision targets of the temporal action modeling TAM and spatial entity modeling SEM losses; through two different classifiers c1(·) and c2(·), the normalized heuristic probability distributions of actions and entities are obtained: in, and They represent the action-inspired probability distribution and entity-inspired probability distribution obtained by the classifier respectively; subscript a m and a n corresponds to the mth action and nth entity in their respective distributions; The TAM loss is calculated as the action-inspired probability distribution derived by the classifier Target distribution obtained with action prompter The cross entropy between: The SEM loss is calculated as the entity-inspired probability distribution obtained by the classifier And the target distribution obtained by the entity prompter The cross entropy between: Use a separate classifier that takes the heuristically enhanced fusion embedding cls As input, the formula is as follows: Will predict the answer Compared with the true label y, the prediction loss is calculated as follows: Combining the prediction loss with the inspiration loss, we get the complete training objective function of the model: Among them, α is a hyperparameter used to balance the weight between action and entity inspiration.

7. The video question answering method with domain-specific fine-grained heuristic prompts according to claim 6, characterized in that: A dynamic gating mechanism is introduced to dynamically adjust the influence of action and entity inspiration according to the input data during the training phase. Specifically: Process the text input through a video language based model to extract text embeddings cls , and passes this representation through a gating network consisting of two multilayer perceptrons, a dropout function, and an activation function: g(t cls )=Sigmoid(MLP2(Dropout(MLP1(t cls )))) (Formula 12) A scalar from 0 to 1 is obtained through the gating mechanism, representing the weight of the action prompt, and the weight of the entity prompt is obtained by subtracting the weight of the action prompt from 1; finally, these weights are applied to the TAM loss and SEM loss to obtain the final combined loss function formula:

Images