A video question-answering method based on a multi-event time relation reasoning framework

Abstract

The application provides a video question answering method based on a multi-event time relationship reasoning framework, which comprises the following steps: preprocessing and encoding video data in a video question answering public data set to obtain segment-level features; preprocessing text data in the video question answering public data set, encoding complex questions, target events and auxiliary events to obtain corresponding text features; inputting the segment-level features and the text features corresponding to the auxiliary events into a Gaussian positioning video module to obtain Gaussian distribution of the target events and the auxiliary events on the video; calculating attention weights of the target events and the auxiliary events on the complex questions, sharing semantic token information and embedding the semantic token information into visual sequences and language sequences to obtain video representations and complex question representations after semantic embedding; performing cross-modal interaction and fusion on the video representations and the complex question representations to obtain global fusion features; and decoding answers based on the global fusion features to obtain final answers. The method effectively improves the quality of video question answering.

Description

Technical Field

[0001] The embodiments of this application relate to the fields of computer vision and natural language processing technology, and in particular to a video question answering method based on a multi-event temporal relationship reasoning framework. Background Technology

[0002] Video question answering (VoQA) tasks involve retrieving relevant spatiotemporal information from a video scene based on a given question, and then generating answers to those questions. In recent years, with the rapid development of visual language tasks, especially since the advent of ChatGPT in 2022, video question answering has become an important tool for evaluating the ability of AI agents to understand human daily behavior. Despite its popularity, video question answering still faces significant challenges. When faced with more complex scenes, multiple events, and long video question answering tasks, models not only need to understand who appears in the video, what objects are present, and what actions are taking place, but more importantly, they must be able to infer the temporal relationships between events in the video based on their understanding of the questions posed.

[0003] Looking back at the development of Video QA, we found (such as) Figure 1 As shown). In terms of video length, the videos consist of short videos under 15 seconds (as shown). Figure 1 From a) to a long video lasting several minutes ( Figure 1 The transition from b1). From the perspective of question type, VideoQA is moving from factual questions (b1). Figure 1 From a and b1) to the problem of temporal reasoning ( Figure 1 The research is developing in the direction of b2). In fact, from the perspective of factual question types, current research methods for short and long videos are essentially the same: matching the question to the most relevant frames or frames of visual information. However, compared to factual questions, temporal reasoning questions contain time-related terms (such as "after" and "before"). As non-visual information, temporal information cannot participate in attention calculations and match the most relevant visual information. Therefore, directly transplanting factual long-video question answering methods to temporal reasoning long-video question answering is not feasible. Unfortunately, current research still uses factual video question answering methods to solve temporal reasoning problems in video question answering tasks.

[0004] A general framework for video QA (such as...) Figure 2The process (as shown) can be summarized into four parts: video encoder, question encoder, cross-modal interaction, and answer decoder. The video encoder represents video information by extracting appearance and motion features at the frame and clip levels, respectively. In recent years, it has been found that object-level visual and semantic features (such as category and attribute labels) are important; these features are typically extracted using pre-trained 2D or 3D neural networks. The language encoder extracts token-level representations, such as GloVe and BERT features. Cross-modal interaction further processes the visual and linguistic sequence data using sequence models (such as RNNs, CNNs, and Transformers) to facilitate cross-modal interaction; this is currently a key area of ​​research. For multiple-choice QA, the answer decoder can be a classifier that selects an answer from the provided multiple choices. For open-ended QA, the answer decoder can be a classifier that selects an answer from a predefined global answer set, or a language generator that generates the answer word by word. The video and language encoders can be pre-trained or recently fine-tuned using classification losses such as hinge loss and cross-entropy loss.

[0005] This general video question-answering framework has the following problems when dealing with complex time-series problems involving multiple events.

[0006] First, the alignment and reasoning processes are opaque. General frameworks are basically based on a holistic understanding of the problem, using language models (such as BERT and Roberta) pre-trained with language encodings to encode the problem. Although this can obtain the implicit temporal structure, when faced with problems with complex temporal and causal relationships, the temporal structural relationships in the problem are opaque, causing problems for subsequent multimodal matching interactions.

[0007] Second, the visual information most relevant to the question obtained through interaction is not perfectly aligned with the question-and-answer pairs. For example... Figure 1 In the example shown in b2, the video clip most relevant to the answer "she placed the sliced ​​cucumber on the bread" is the red section. However, the question also includes another action, "slicing the cucumber," which is not included in the red video clip. Obtaining visual clips that don't perfectly match the ground-truth labels in a video question-answering task can negatively impact the model's training accuracy. This is because the model might learn incorrect feature representations or biased patterns, leading to performance degradation. Summary of the Invention

[0008] To address the aforementioned technical issues, embodiments of this application propose a video question-answering method based on a multi-event temporal relationship reasoning framework. Drawing upon human thinking and reasoning strategies, this method uses an event-level language parsing module to extract clauses, sub-questions, and temporal relationships related to individual events. Complex questions are broken down into simple, independent events, and a video localization mechanism is introduced to locate independent events to video segments. This helps the model confirm and reason based on factual evidence in the video to answer questions, effectively improving the quality of video question answering.

[0009] To achieve the above objectives, embodiments of this application propose a video question-answering method based on a multi-event temporal relationship reasoning framework. The method includes: preprocessing video data from a publicly available video question-answering dataset and inputting the preprocessed video data into a multi-stream video encoder for encoding to obtain segment-level features; preprocessing text data from the publicly available video question-answering dataset, extracting several independent target events and related auxiliary events from a complex question containing multiple events using an event-level language parsing module, and then inputting them together into the language encoder for encoding to obtain corresponding text features; inputting the segment-level features and the text features corresponding to the auxiliary events into a video Gaussian localization module to obtain the Gaussian distribution of the target events on the video and the distribution of the auxiliary events on the video. The Gaussian distribution of the target event and auxiliary events is used to calculate the attention weights of the target event and auxiliary events on the complex problem. A multi-event semantic lexical embedding module is used, combining the Gaussian distributions of the target event and auxiliary events in the video, and setting two different event-level tokens as shared semantic lexical information. These tokens are embedded into the visual sequence and the language sequence to obtain the semantically embedded video representation and the semantically embedded complex problem representation. The semantically embedded video representation and the semantically embedded complex problem representation are then input into the cross-modal interaction module for cross-modal interaction and fusion to obtain global fusion features. These global fusion features are then input into the answer decoder module, which calculates the similarity between the global fusion features and candidate answers, and selects the candidate answer with the highest similarity as the final answer.

[0010] To achieve the above objectives, embodiments of this application also propose an electronic device comprising: a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a video question-answering method based on a multi-event temporal relationship reasoning framework as described above.

[0011] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a video question-answering method based on a multi-event temporal relationship reasoning framework as described above.

[0012] Optionally, the video data in the publicly available video question-and-answer dataset is preprocessed, and the preprocessed video data is input into a multi-stream video encoder for encoding to obtain segment-level features. This includes: sparsely sampling the video data in the publicly available video question-and-answer dataset according to a preset number of frames to obtain multiple video segments of the same length, and performing preprocessing on each video segment, including high-confidence region selection; inputting the preprocessed video segments into the multi-stream video encoder to model the video features to obtain segment-level features for each video segment; wherein, the set of segment-level features for each video segment is denoted as . , , For the first Fragment-level features of a video segment This represents the total number of video clips.

[0013] Optionally, the text data in the video question-answering public dataset is preprocessed. An event-level language parsing module extracts several independent target events and related auxiliary events from the complex question containing multiple events. These are then input into the language encoder for encoding to obtain the corresponding text features. This includes: preprocessing the text data in the video question-answering public dataset, including typo correction, and inputting the preprocessed text data into the event-level language parsing module. The event-level language parsing module performs event-level semantic parsing on the preprocessed text data. Through the event-level language parsing module, the complex question containing multiple events is decomposed into several clauses representing independent events, and related time information is obtained. The independent events corresponding to the declarative sentences in the clauses are named auxiliary events for subsequent Gaussian localization, and the independent events corresponding to the interrogative sentences in the clauses are named target events, serving as the actual questions to be answered. Here, the complex question, target events, and auxiliary events are denoted as... , , Based on time information, determine Compared to The time relationship; among them, Compared to Temporal relationships include three types: occurring first, occurring later, and overlapping; , , These are input together into the language encoder for encoding to obtain good contextual word representations, thereby obtaining the corresponding text features; among them, , , The corresponding text features are denoted as follows: , , .

[0014] Optionally, the fragment-level features and the text features corresponding to the auxiliary events are input into the video Gaussian localization module to obtain the Gaussian distribution of the target event and the Gaussian distribution of the auxiliary events in the video, including: Based on Transformer encoder and Transformer decoder and Multimodal interaction is performed to obtain fusion features that combine semantic and visual information. ; use Prediction with Sigmoid activation function The center of the Gaussian distribution in the video and width and based on and get Gaussian mask , , express The Middle The weight of each video segment; based on Compared to Due to time constraints, it is estimated The center of the Gaussian distribution in the video and width and based on and get Gaussian mask , , express The Middle The weight of each video segment; Among them, if Compared to If the temporal relationship is that it occurred first, then , ; like Compared to If the time relationship is that it occurs later, then , ; like Compared to If the time relationship is overlapping, then , .

[0015] Optionally, based on the Transformer encoder and Transformer decoder... and Multimodal interaction is performed to obtain fusion features that combine semantic and visual information. This can be achieved through the following formula: ; in, and These are the Transformer encoder and Transformer decoder, respectively. To hide the state dimension, This is a fusion feature that combines the segment features of all video clips and the text features of auxiliary events; use Prediction with Sigmoid activation function The center of the Gaussian distribution in the video and width This can be achieved through the following formula: , ; in, It is the Sigmoid activation function. It is a fully connected layer. ; based on and get Gaussian mask ,based on and get Gaussian mask This can be achieved using the following formulas: ; ; in, Hyperparameters for controlling the width of Gaussian curves.

[0016] Optionally, attention weights for the target event and auxiliary events on the complex problem are calculated. Using a multi-event semantic lexical embedding module, and combining the Gaussian distributions of the target event and auxiliary events in the video, two different event-level tokens are set as shared semantic lexical information and embedded into the visual and linguistic sequences to obtain semantically embedded video representations and semantically embedded complex problem representations, including: calculate and exist Attention weights and ; Utilizing a multi-event semantic lexical embedding module, combined with and Set two different event-level tokens and As shared semantic lexical information; among which... for The corresponding lexical embedding, for Corresponding lexical embeddings; based on and ,Will and Embedded into In this process, a complex problem representation is obtained after semantic embedding. ; based on and ,Will and Embedded into In the process, the semantically embedded video representation is obtained. .

[0017] Optionally, based on and ,Will and Embedded into In this process, a complex problem representation is obtained after semantic embedding. ,based on and ,Will and Embedded into In the process, the semantically embedded video representation is obtained. This can be achieved through the following formula: ; .

[0018] Optionally, the semantically embedded video representation and the semantically embedded complex problem representation are input into the cross-modal interaction module for cross-modal interaction and fusion to obtain global fusion features, including: Using the following formula, merge into In the middle, the fusion representation is obtained. : , ; in, Indicates the total number of lexical units; right After processing by a Transformer, average pooling is then performed to obtain the global fused features.

[0019] This application proposes a video question answering method based on a multi-event temporal relationship reasoning framework. Addressing the shortcomings of opaque and inaccurate multimodal information alignment and reasoning processes in video question answering tasks involving complex questions with multiple events, this method fully draws on human thinking and reasoning strategies and has the following beneficial effects compared with traditional methods.

[0020] First, it achieves structured analysis of complex temporal problems, making the temporal reasoning process transparent and traceable. This application uses an event-level language parsing module to decompose complex problems involving multiple events into independent target events and auxiliary events, and explicitly extracts the temporal relationships and time intervals between events. It abandons the implicit encoding of the entire problem by traditional methods, making the temporal reasoning logic of the model clear and interpretable instead of a black box, and completely solves the defects of traditional methods such as opaque temporal relationships and confusion of multiple events interfering with subsequent multimodal interactions.

[0021] Second, it accurately performs event-level video localization, ensuring the accuracy of multimodal information alignment. This application introduces a Gaussian video localization mechanism. Based on the characteristic of continuous distribution of events in videos, it generates Gaussian distribution masks for auxiliary events and target events, accurately locating video segments that match each event. This allows the model to rely entirely on real visual evidence from the video for reasoning, rather than relying on language model biases to generate answers. This avoids erroneous feature learning caused by misalignment between visual segments and question-answer pairs, fundamentally improving the accuracy of multimodal matching and ensuring the effectiveness of model learning.

[0022] Third, by explicitly embedding multi-event semantic tokens, this application strengthens multimodal event-level relational interactions. It sets up two types of dedicated learnable semantic tokens—target events and auxiliary events—and embeds them as shared semantic information into visual and linguistic sequences. This explicitly labels the event types in multimodal data, addressing the problem that traditional attention mechanisms cannot handle non-visual information such as time-related words and are prone to confusing multi-event dependencies. This builds an event-level bridge for cross-modal interaction, ensuring the model fully understands multi-event information in video and language, allowing multimodal interaction to focus on core events relevant to the problem.

[0023] Fourth, it breaks through the limitations of long-video, multi-event scenarios, significantly improving the performance of temporal reasoning question answering. The new event-level multimodal alignment and reasoning paradigm constructed in this application is specifically adapted to question answering scenarios involving long videos, multiple events, and complex temporal relationships. It effectively filters out interference from irrelevant video information, allowing the model to output answers based on rigorous temporal logic and real visual evidence, greatly improving the accuracy of answers to complex temporal reasoning questions, and breaking through the technical bottleneck that traditional methods are only applicable to short videos.

[0024] Fifth, it possesses excellent versatility and scalability. The event-level language parsing, Gaussian video localization, and multi-event semantic token embedding modules of this application are all independent pluggable units, which can seamlessly adapt to the core modules of existing mainstream video question answering frameworks, such as video encoding, question encoding, cross-modal interaction, and answer decoding, without the need to reconstruct the overall framework. This effectively reduces the cost of technology implementation and expands the application scenarios and adaptability of video question answering methods. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. The following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0026] Figure 1 This is a diagram illustrating the evolution of video Q&A. Figure 2 This is a schematic diagram of the mainstream video question-answering framework; Figure 3 This is a flowchart of a video question-answering method based on a multi-event temporal relationship reasoning framework provided in one embodiment of this application; Figure 4 This is a schematic diagram of event-level language parsing provided in one embodiment of this application; Figure 5 This is a schematic diagram of timing relationships and time intervals provided in one embodiment of this application; Figure 6 This is a schematic diagram illustrating the location and estimation of auxiliary and target events provided in one embodiment of this application; Figure 7 This is a schematic diagram illustrating the weight distribution of the target event and auxiliary events on the video sequence and the question sequence, respectively, in one embodiment of this application; Figure 8 This is a schematic diagram of the structure of a video question-answering framework provided in one embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0028] One embodiment of this application proposes a video question answering method based on a multi-event temporal relationship reasoning framework. The implementation details of the video question answering method based on a multi-event temporal relationship reasoning framework proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0029] The specific process of the video question-answering method based on a multi-event temporal relationship reasoning framework proposed in this embodiment can be described as follows: Figure 3 As shown, it includes: Step 101: Preprocess the video data in the video question-and-answer public dataset, and input the preprocessed video data into the multi-stream video encoder for encoding to obtain segment-level features.

[0030] In its specific implementation, the video question-answering framework of this application (such as...) Figure 8 As shown, we need to approach this from two aspects: video and text. On the video side, we need to preprocess the video data in the video question-and-answer public dataset and input the preprocessed video data into a multi-stream video encoder for encoding to obtain segment-level features.

[0031] In one example, the publicly available video question-answering dataset used could be the NExT-QA dataset, with the video data in the NExT-QA dataset processed according to a preset frame rate. Sparse sampling is performed on 32 frames to obtain multiple samples of the same length (length denoted as...). , The dataset contains eight video clips, and each clip undergoes preprocessing, including high-confidence region selection. High-confidence region selection is achieved using an object detection model, specifically a Faster R-CNN with a ResNet-101 backbone pre-trained on the Visual Genome dataset. This model detects and selects the top five regions with high confidence by default.

[0032] Next, the preprocessed video segments are input into a multi-stream video encoder to model the video features, obtaining segment-level features for each video segment. The set of segment-level features for each video segment is denoted as . , , For the first Fragment-level features of a video segment This represents the total number of video clips.

[0033] In one example, the multi-stream video encoder can be a Dynamic Graphics Transformer (DGT) module.

[0034] Step 102: The text data in the video question-answering public dataset is preprocessed. The event-level language parsing module extracts several independent target events and related auxiliary events from the complex question containing multiple events. These are then input into the language encoder for encoding to obtain the corresponding text features.

[0035] In the specific implementation, regarding the text, the text data in the video question-and-answer public dataset is preprocessed. The event-level language parsing module extracts several independent target events and related auxiliary events from the complex question containing multiple events. These are then input into the language encoder for encoding to obtain the corresponding text features.

[0036] The current mainstream approach treats the question text as a whole and inputs it into the network, ignoring the decomposability of complex questions composed of multiple events. However, understanding each sub-question helps the model to semantically parse the original complex question. Based on the characteristics and grammatical rules of the NExT-QA dataset, this embodiment designs a simple event-level sentence parsing method that can decompose complex sentences containing multiple events into clauses of a single independent event.

[0037] In one example, this embodiment preprocesses the text data from the video question-answering public dataset, including typo correction, and then inputs the preprocessed text data into an event-level language parsing module. This module performs event-level semantic parsing on the preprocessed text data. Through the event-level language parsing module, complex questions containing multiple events are decomposed into several clauses representing independent events, and relevant time information is obtained. The independent events corresponding to the declarative sentences in the clauses are named Auxiliary Events for subsequent Gaussian localization, and the independent events corresponding to the interrogative sentences in the clauses are named Target Events, which serve as the actual questions to be answered. The complex question, target event, and auxiliary event are denoted as follows: , , (like Figure 4(As shown).

[0038] In one example, this embodiment parses out the time information. Compared to Due to time constraints. Compared to The temporal relationships include three types: before, after, and overlap, primarily used for inferring the location information of events. The three time intervals are START, MIDDLE, and FINISH, such as... Figure 5 As shown, this is mainly to facilitate the precise location of the time range for queries in video problems, thereby minimizing calculations involving irrelevant information.

[0039] refer to Figure 4 The original problem, "what does the girl in white do after bending down in the middle," is decomposed into the auxiliary event "the girl in white do bending down" and the target event "what does the girl in white do," with the temporal relationship being AFTER and the time interval being MIDDLE.

[0040] In one example, , , These are input together into the language encoder for encoding to obtain good contextual word representations, thereby obtaining the corresponding text features. , , .

[0041] Step 103: Input the fragment-level features and the text features corresponding to the auxiliary events into the video Gaussian localization module to obtain the Gaussian distribution of the target event and the Gaussian distribution of the auxiliary events in the video.

[0042] In practical implementation, since an event usually includes a beginning, climax and end, the location of an event on a video should be a series of consecutive clips. Based on this characteristic, this embodiment uses Gaussian mask to represent the inherent temporal structure of the event. That is, the clip-level features and the text features corresponding to the auxiliary events are input into the video Gaussian localization module to obtain the Gaussian distribution of the target event and the Gaussian distribution of the auxiliary events on the video.

[0043] In one example, the process of locating and estimating auxiliary and target events is as follows: Figure 6 As shown.

[0044] The first step is the localization of auxiliary events. This embodiment is based on the Transformer encoder and Transformer decoder. and Multimodal interaction is performed to obtain fusion features that combine semantic and visual information. Then utilize Prediction with Sigmoid activation function The center of the Gaussian distribution in the video and width and based on and get Gaussian mask , , express The Middle The weight of each video segment.

[0045] Based on Transformer encoder and Transformer decoder and Multimodal interaction is performed to obtain fusion features that combine semantic and visual information. This can be achieved through the following formula: ; in, and These are the Transformer encoder and Transformer decoder, respectively. To hide the state dimension, This is a fusion feature that combines the segment features of all video clips with the text features of auxiliary events.

[0046] use Prediction with Sigmoid activation function The center of the Gaussian distribution in the video and width This can be achieved through the following formula: , ; in, It is the Sigmoid activation function. It is a fully connected layer. .

[0047] based on and get Gaussian mask This can be achieved through the following formula: ; in, Hyperparameters for controlling the width of Gaussian curves.

[0048] Next is the estimation of the target event. The target event is an interrogative statement, unlike auxiliary events which can be directly located in the video segment. Since the exact scope of the target event as the real problem is unclear, estimation is necessary. This embodiment is based on... Compared to Due to time constraints, it is estimated The center of the Gaussian distribution in the video and width and based on and get Gaussian mask , , express The Middle The weight of each video segment.

[0049] like Compared to If the time relationship is that it happened first, then... It must happen On the left side, , .

[0050] like Compared to If the time relationship is that it happens later, then It must happen On the right side, , .

[0051] like Compared to The time relationships overlap, then and If they occur simultaneously, then , .

[0052] based on and get Gaussian mask This can be achieved through the following formula: .

[0053] Step 104: Calculate the attention weights of the target event and auxiliary events on the complex problem. Using the multi-event semantic lexical embedding module, combine the Gaussian distribution of the target event and the Gaussian distribution of the auxiliary events on the video, set two different event-level tokens as shared semantic lexical information, and embed them into the visual sequence and the language sequence to obtain the semantically embedded video representation and the semantically embedded complex problem representation.

[0054] In practical implementation, the most important aspect of video question answering tasks is multimodal interaction alignment. Traditional methods mainly fall into two categories: one is to directly implement it using a multimodal Transformer, and the other is to directly implement it using a similarity calculation module. Both methods are based on attention mechanisms and do not provide transparent parsing of complex sentences. Two events with a temporal relationship inevitably have a strong dependency, which inevitably leads to confusion between the two events during multimodal attention calculation. Furthermore, the attention calculation process does not involve logical reasoning, so attention alone cannot handle non-visual words such as those related to time relationships. To enable the model to perform multimodal alignment based on events, this embodiment calculates the attention weights of the target event and auxiliary events on the complex question. Using a multi-event semantic lexical embedding module, combining the Gaussian distribution of the target event and the auxiliary event on the video, two different event-level tokens are set as shared semantic lexical information and embedded into the visual sequence and language sequence to obtain the semantically embedded video representation and the semantically embedded complex question representation.

[0055] In the multi-event semantic lexical embedding process, first calculate and exist Attention weights and , and like Figure 7 As shown. Next, the multi-event semantic lexical embedding module will be used in conjunction with... and Set two different event-level tokens and As shared semantic lexical information; among which... for The corresponding lexical embedding, for Corresponding lexical embeddings. (Based on...) and ,Will and Embedded into In this process, a complex problem representation is obtained after semantic embedding. Finally, based on and ,Will and Embedded into In the process, the semantically embedded video representation is obtained. .

[0056] Multi-event semantic lexical embedding can explicitly label the event types corresponding to multimodalities. As a bridge in the subsequent multimodal interaction matching calculation process, it allows complex multi-event problems and long multi-event videos to focus on two events related to the problem at the same time, providing complete visual evidence to support the answer for the subsequent multimodal reasoning module.

[0057] In one example, based on and ,Will and Embedded into In this process, a complex problem representation is obtained after semantic embedding. ,based on and ,Will and Embedded into In the process, the semantically embedded video representation is obtained. This can be achieved through the following formula: , .

[0058] Step 105: Input the semantically embedded video representation and the semantically embedded complex problem representation into the cross-modal interaction module for cross-modal interaction and fusion to obtain global fusion features.

[0059] Step 106: Input the global fusion features into the answer decoder module. The answer decoder module calculates the similarity between the global fusion features and the candidate answers, and selects the candidate answer with the highest similarity as the final answer.

[0060] In practical implementation, after obtaining the semantically embedded video representation and the semantically embedded complex question representation, these can be input into the cross-modal interaction module for cross-modal interaction and fusion to obtain global fusion features. These global fusion features are then input into the answer decoder module (see reference). Figure 8 The answer decoder module calculates the similarity between global fusion features and candidate answers, and selects the candidate answer with the highest similarity as the final answer.

[0061] In one example, the semantically embedded video representation and the semantically embedded complex problem representation are input into the cross-modal interaction module for cross-modal interaction and fusion. To obtain the global fusion features, the following formula is first used: merge into In the middle, the fusion representation is obtained. : , ;in, This indicates the total number of lexical units. Subsequently, regarding... After processing by a Transformer, average pooling is then performed to obtain the global fused features.

[0062] This embodiment proposes a video question answering method based on a multi-event temporal relationship reasoning framework. Addressing the shortcomings of opaque and inaccurate multimodal information alignment and reasoning processes in video question answering tasks involving complex questions with multiple events, this method fully draws on human thinking and reasoning strategies and has the following beneficial effects compared to traditional methods.

[0063] First, it achieves structured analysis of complex temporal problems, making the temporal reasoning process transparent and traceable. This embodiment uses an event-level language parsing module to decompose complex problems containing multiple events into independent target events and related auxiliary events, and explicitly extracts the temporal relationships and time intervals between events. It abandons the implicit encoding of the entire problem in traditional methods, making the temporal reasoning logic of the model clear and interpretable instead of a black box. This completely solves the defects of traditional methods, such as opaque temporal relationships and confusion caused by multiple events interfering with subsequent multimodal interactions.

[0064] Second, it accurately completes event-level video localization, ensuring the accuracy of multimodal information alignment. This embodiment introduces a Gaussian video localization mechanism. Based on the characteristic of continuous distribution of events in videos, it generates Gaussian distribution masks for auxiliary events and target events, accurately locating video segments that match each event. This allows the model to rely entirely on real visual evidence from the video for reasoning, rather than relying on language model biases to generate answers. This avoids erroneous feature learning caused by misalignment between visual segments and question-answer pairs, fundamentally improving the accuracy of multimodal matching and ensuring the effectiveness of model learning.

[0065] Third, by explicitly embedding multi-event semantic tokens, multimodal event-level associative interactions are enhanced. This embodiment sets up two types of dedicated learnable semantic tokens: target events and auxiliary events. These tokens are embedded as shared semantic information into visual and linguistic sequences, explicitly marking event types in multimodal data. This addresses the problem that traditional attention mechanisms cannot handle non-visual information such as time-related words and are prone to confusing multi-event dependencies. It builds an event-level bridge for cross-modal interaction, ensuring that the model fully understands multi-event information in video and language, allowing multimodal interaction to focus on core events related to the problem.

[0066] Fourth, it overcomes the limitations of long-video, multi-event scenarios, significantly improving the performance of temporal reasoning question answering. The new paradigm of event-level multimodal alignment and reasoning constructed in this embodiment is suitable for question answering scenarios involving long videos, multiple events, and complex temporal relationships. It effectively filters out interference from irrelevant video information, allowing the model to output answers based on rigorous temporal logic and real visual evidence, greatly improving the accuracy of answers to complex temporal reasoning questions, and breaking through the technical bottleneck that traditional methods are only applicable to short videos.

[0067] Fifth, it possesses excellent versatility and scalability. The event-level language parsing, Gaussian video localization, and multi-event semantic token embedding modules in this embodiment are all independent pluggable units, which can seamlessly adapt to the core modules of existing mainstream video question answering frameworks, such as video encoding, question encoding, cross-modal interaction, and answer decoding, without the need to reconstruct the overall framework. This effectively reduces the cost of technology implementation and expands the application scenarios and adaptability of video question answering methods.

[0068] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0069] Another embodiment of this application provides an electronic device, such as Figure 9 As shown, it includes a processor 201 and a memory 202. The memory 202 stores instructions that the processor 201 can execute. When the processor 201 is configured to execute the instructions, the electronic device can implement a video question answering method based on a multi-event temporal relationship reasoning framework as described in the above method embodiment.

[0070] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0071] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0072] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a video question-answering method based on a multi-event temporal relationship reasoning framework as described in the above method embodiments.

[0073] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0074] It will be understood by those skilled in the art that the above embodiments are specific embodiments for implementing this application, and various changes in form and detail can be made in practical applications without departing from the scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A video question-answering method based on a multi-event temporal relationship reasoning framework, characterized in that, include: The video data in the video question-and-answer public dataset is preprocessed, and the preprocessed video data is input into a multi-stream video encoder for encoding to obtain segment-level features; The text data in the video question-answering public dataset is preprocessed. The event-level language parsing module extracts several independent target events and related auxiliary events from the complex question containing multiple events. These are then input into the language encoder for encoding to obtain the corresponding text features. The fragment-level features and the text features corresponding to the auxiliary events are input into the video Gaussian localization module to obtain the Gaussian distribution of the target event and the Gaussian distribution of the auxiliary events in the video. Calculate the attention weights of the target event and auxiliary events on the complex problem, utilize the multi-event semantic lexical embedding module, combine the Gaussian distribution of the target event on the video and the Gaussian distribution of the auxiliary events on the video, set two different event-level tokens as shared semantic lexical information, embed them into the visual sequence and the language sequence, and obtain the semantically embedded video representation and the semantically embedded complex problem representation. The semantically embedded video representation and the semantically embedded complex problem representation are input into the cross-modal interaction module for cross-modal interaction and fusion to obtain global fusion features; The global fusion features are input into the answer decoder module, which calculates the similarity between the global fusion features and the candidate answers, and selects the candidate answer with the highest similarity as the final answer.

2. The video question-answering method based on a multi-event temporal relationship reasoning framework as described in claim 1, characterized in that, The video data in the publicly available video question-answering dataset is preprocessed, and the preprocessed video data is then input into a multi-stream video encoder for encoding to obtain segment-level features, including: The video data in the video question-and-answer public dataset is sparsely sampled according to a preset number of frames to obtain multiple video segments of the same length, and each video segment is preprocessed, including the selection of high-confidence regions. The preprocessed video segments are input into a multi-stream video encoder to model the video features and obtain the segment-level features of each video segment. The set of segment-level features for each video segment is denoted as . , , For the first Fragment-level features of a video segment This represents the total number of video clips.

3. The video question-answering method based on a multi-event temporal relationship reasoning framework according to claim 2, characterized in that, The text data in the video question-answering public dataset is preprocessed. An event-level language parsing module extracts several independent target events and related auxiliary events from complex questions containing multiple events. These are then input into the language encoder for encoding to obtain the corresponding text features, including: The text data in the video question-and-answer public dataset is preprocessed, including typo correction, and then the preprocessed text data is input into the event-level language parsing module, which performs event-level semantic parsing on the preprocessed text data. The event-level language parsing module decomposes complex problems containing multiple events into several clauses representing independent events and obtains relevant time information. The independent events corresponding to the declarative sentences in the clauses are named auxiliary events for subsequent Gaussian localization, while the independent events corresponding to the interrogative sentences in the clauses are named target events, serving as the actual questions to be answered. The complex question, target event, and auxiliary events are denoted as follows: , , ; Based on time information, determine Compared to The time relationship; among them, Compared to Temporal relationships include three types: those that occur first, those that occur later, and those that overlap. Will , , These are input together into the language encoder for encoding to obtain good contextual word representations, thereby obtaining the corresponding text features; among them, , , The corresponding text features are denoted as follows: , , .

4. The video question-answering method based on a multi-event temporal relationship reasoning framework as described in claim 3, characterized in that, The fragment-level features and the text features corresponding to the auxiliary events are input into the video Gaussian localization module to obtain the Gaussian distribution of the target event and the Gaussian distribution of the auxiliary events in the video, including: Based on Transformer encoder and Transformer decoder and Multimodal interaction is performed to obtain fusion features that combine semantic and visual information. ; use Prediction with Sigmoid activation function The center of the Gaussian distribution in the video and width and based on and get Gaussian mask , , express The Middle The weight of each video segment; based on Compared to Due to time constraints, it is estimated The center of the Gaussian distribution in the video and width and based on and get Gaussian mask , , express The Middle The weight of each video segment; Among them, if Compared to If the temporal relationship is that it occurred first, then... , ; like Compared to If the time relationship is that it occurs later, then , ; like Compared to If the time relationship is overlapping, then , .

5. The video question-answering method based on a multi-event temporal relationship reasoning framework as described in claim 4, characterized in that, Based on Transformer encoder and Transformer decoder and Multimodal interaction is performed to obtain fusion features that combine semantic and visual information. This can be achieved through the following formula: ； in, and These are the Transformer encoder and Transformer decoder, respectively. To hide the state dimension, This is a fusion feature that combines the segment features of all video clips and the text features of auxiliary events; use Prediction with Sigmoid activation function The center of the Gaussian distribution in the video and width This can be achieved through the following formula: ， ； in, It is the Sigmoid activation function. It is a fully connected layer. ; based on and get Gaussian mask ,based on and get Gaussian mask This can be achieved using the following formulas: ； ； in, Hyperparameters for controlling the width of Gaussian curves.

6. The video question-answering method based on a multi-event temporal relationship reasoning framework according to claim 4, characterized in that, The attention weights of the target event and auxiliary events on the complex problem are calculated. Using a multi-event semantic lexical embedding module, the Gaussian distributions of the target event and auxiliary events in the video are combined. Two different event-level tokens are set as shared semantic lexical information and embedded into the visual and linguistic sequences to obtain semantically embedded video representations and semantically embedded representations of the complex problem, including: calculate and exist Attention weights and ; Utilizing a multi-event semantic lexical embedding module, combined with and Set two different event-level tokens and As shared semantic lexical information; among which... for The corresponding lexical embedding, for Corresponding lexical embeddings; based on and ,Will and Embedded into In this process, a complex problem representation is obtained after semantic embedding. ; based on and ,Will and Embedded into In the process, the semantically embedded video representation is obtained. .

7. A video question-answering method based on a multi-event temporal relationship reasoning framework according to claim 6, characterized in that, based on and ,Will and Embedded into In this process, a complex problem representation is obtained after semantic embedding. ,based on and ,Will and Embedded into In the process, the semantically embedded video representation is obtained. This can be achieved through the following formula: ； 。 8. A video question-answering method based on a multi-event temporal relationship reasoning framework according to claim 6, characterized in that, The semantically embedded video representation and the semantically embedded complex problem representation are input into the cross-modal interaction module for cross-modal interaction and fusion to obtain global fusion features, including: Using the following formula, Merge into In the middle, the fusion representation is obtained. : ， ； in, Indicates the total number of lexical units; right After processing by a Transformer, average pooling is then performed to obtain the global fused features.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a video question answering method based on a multi-event temporal relationship reasoning framework as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a video question answering method based on a multi-event temporal relationship reasoning framework as described in any one of claims 1 to 8.

Images