A method and system for event hub enhancement for video question answering

Abstract

The application provides a video question-answering event center enhancement method and system, and relates to the fields of artificial intelligence, multi-modal video understanding, video question-answering and weakly supervised time sequence positioning. The application generates video-level and frame-level time sequence perception description for an input video, synthesizes augmented triple data containing time span, question and answer from continuous event segments, cleans the synthesized triple through event segment-question alignment and video text correlation sorting, automatically generates pseudo event span for original video question-answering samples without time span, forms an enhanced training set, designs a weakly supervised key event positioning training strategy based on the pseudo event span, and guides a video question-answering model to improve key frame confidence and suppress irrelevant frame interference. The application solves the problems of existing video question-answering data labeling scarcity, lack of key event time supervision and model answer basis untrustworthiness.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, multimodal video understanding, video question answering and weakly supervised temporal localization, and in particular to an event-centric enhancement method and system for video question answering. Background Technology

[0002] Video question answering tasks require models to simultaneously understand natural language questions and video content, and infer the correct answer that matches the question from the video. Compared to static image question answering, video question answering not only needs to identify objects, scenes, and actions, but also needs to understand the temporal and causal relationships of events, as well as the correlations between multiple events. Therefore, the model's ability to locate key event segments relevant to the question directly affects the accuracy and credibility of the final answer.

[0003] Existing video question-answering datasets typically only provide video, question, and answer annotations, lacking supervision on the start and end times of key events or keyframes corresponding to the questions. Furthermore, due to the high cost of manual annotation, a single video often contains only a small number of question-answer pairs, making it difficult to cover the rich action details, event transitions, causal clues, and temporal structure within the video. For complex videos containing multiple sub-events, limited question-answer annotations cause the model to learn only shallow semantic relationships, failing to fully model the underlying event-level semantics of the video.

[0004] To enhance multimodal semantic understanding capabilities, existing methods typically rely on large-scale visual language pre-training, hierarchical attention mechanisms, graph-structured reasoning mechanisms, or salient frame ranking mechanisms. However, there are distributional differences between large-scale pre-training corpora and downstream video question-answering data, making it difficult for direct pre-training to fully learn the temporal and causal information in specific videos. While salient frame ranking methods can select frames relevant to the question to some extent, the lack of event span supervision means the model may still focus on irrelevant backgrounds or local noise, leading to unstable answer predictions.

[0005] Therefore, how to automatically expand the event semantic coverage of video question answering training data without relying on expensive manual event span annotation, and provide usable weak temporal supervision for video question answering models to improve their key event localization capabilities and answer prediction reliability, has become an urgent problem to be solved in the field of video question answering technology. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention provides an event-centric enhancement method and system for video question answering. This method solves the problems of scarce question and answer annotations, insufficient event-level semantic coverage, lack of supervision over the time span of key events, and insufficient reliability of answer prediction in existing video question answering methods. It significantly improves the semantic coverage, temporal awareness, key event localization, and answer prediction reliability in complex multi-event videos.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: an event-centric enhancement method for video question answering, comprising the following steps: S1. Use a visual descriptor to perform time-aware description generation on the video frame sequence to obtain frame-level descriptions corresponding to each video frame, and generate event-centric SQA triples based on the frame-level descriptions. S2. Based on event fragment-question-answer alignment and video text relevance ranking, the event center SQA triplet is cleaned to obtain a high-quality event center triplet set. S3. Using a video moment retrieval tool, generate corresponding pseudo-event spans for the original question and answer, and combine them with the original question and answer to form the original pseudo-SQA triples; merge the set of high-quality event center triples with the original pseudo-SQA triples to construct an enhanced video question answering training set. S4. Train the video question answering model using the enhanced video question answering training set, and construct a weakly supervised key event localization loss using pseudo-event span during the training process. This guides the video question answering model to capture key event fragments related to the question while predicting the answer, thus completing the enhanced processing of the event.

[0008] Further, S1 includes: The input video is sampled to obtain several video frames arranged in chronological order; A video-level global description is obtained by performing time-aware description generation on a video frame sequence using a visual descriptor. ; For each video frame, associate the video frame with the video-level global description. All data is input into the video descriptor to obtain the frame-level description corresponding to each video frame. Among them, frequency level global description As a time context constraint; Based on the frame-level description sequence, at least one continuous time span is randomly sampled. ,in, and These represent the start frame index and the end frame index, respectively, and satisfy the preset length constraint. , and These represent the minimum and maximum preset lengths, respectively. Extracting continuous time span Within the frame-level description, a sequence of continuous event fragment descriptions is obtained. ; Describing a sequence of consecutive event segments Input a question-answer synthesizer to generate multiple question-answer pairs that address the overall event content over a continuous time span; continuous time span By concatenating these with each question-answer pair, we obtain the event center SQA triplet.

[0009] Furthermore, S2 includes: For candidate triplet Based on its time span, the corresponding event video segments are cropped from the input video. ;in, and These represent the start frame index and the end frame index, respectively. This indicates that the question and answer are correct. Video clips of the incident And question-answer pairs Input event snippet - Question and answer aligner, determine event video snippets Can the corresponding answer be inferred? If the result is misalignment, then the candidate triplet is discarded. If the result is alignment, then retain the candidate triplet. ; For the retained aligned triples, the event video segments are calculated using a video-text relevance sorter. Question-answer pair Cross-modal correlation scores between them; Based on cross-modal relevance scores, top-ranked videos are retained for each video. N The set of high-quality event-centric triples is obtained by finding the triples with the highest correlation. .

[0010] Furthermore, S3 includes: Input video, original question and the original answer Enter the video timer to search for question-and-answer pairs. The most semantically relevant consecutive video segments are used to obtain the pseudo-start frame index. and pseudo-end frame index Among them, the pseudo-start frame index and pseudo-end frame index The span of pseudo-events; Connecting the span of pseudo-events with the original question and the original answer Combined, forming the original pseudo-SQA triplet. ; Let all the original pseudo-SQA triples be denoted as and the high-quality event center triplet set Merging them yields an enhanced video question-answering training set. .

[0011] Furthermore, the construction process of the weakly supervised key event localization loss includes: The multimodal fusion video representation of the video question answering model is input into the lightweight frame scoring head to obtain the frame importance score corresponding to each video frame; Based on frame importance scores, the top K video frames with the highest significance scores are selected, and the Top-ranked frames are further grouped according to the pseudo-event span of the current sample. K The frame index is divided into a set of frames within the pseudo-event span and a set of indices outside the pseudo-event span; A sampling window is constructed with the center frame of the pseudo-event span as the center, and the set of frames within the span is removed from the sampling window to obtain the set of potential positive sample frames; video frames that do not fall into the sampling window and do not belong to the set of frames within the span are identified as the set of negative sample frames. The loss for locating key events under weak supervision is calculated based on the set of potential positive sample frames and the set of negative sample frames.

[0012] Furthermore, the objective function of the video question-answering model is expressed as follows:

[0013]

[0014] in, This represents the objective function of the video question-answering model. This represents the loss term for predicting video question-and-answer answers. This indicates the identification of loss items in key events where supervision is weak. express The weight hyperparameters, Represents the set of potential positive sample frames. Indicates the total number of video frames. Indicates the temperature coefficient. Indicates the first One potential positive sample frame, Indicates the first One video frame, Indicates the first The significance score of each potential positive sample frame. Indicates the first The saliency score of each video frame.

[0015] The present invention also provides an event center enhancement system for video question answering, comprising: The first processing module is used to perform time-aware description generation on the video frame sequence using a visual descriptor, obtain the frame-level description corresponding to each video frame, and generate an event-centric SQA triple based on the frame-level description. The second processing module is used to clean the event center SQA triples based on event fragment-question-answer alignment and video text relevance sorting to obtain a high-quality event center triple set. The third processing module is used to generate corresponding pseudo-event spans for the original question and answer using a video moment retrieval device, and combine them with the original question and answer to form the original pseudo-SQA triples; the synthesized high-quality event center triples set is merged with the original pseudo-SQA triples to construct an enhanced video question answering training set; The fourth processing module is used to train the video question answering model using the enhanced video question answering training set. During the training process, it constructs a weakly supervised key event localization loss using pseudo-event spans, guiding the video question answering model to capture key event fragments related to the question while predicting the answer, thus completing the enhanced processing of the event.

[0016] The present invention also provides an electronic device including a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the electronic device to perform the various steps of the event-centric enhancement method for video question answering.

[0017] The present invention also provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the various steps of the event-centric enhancement method for video question answering.

[0018] The core idea of ​​this invention is to expand video question-answering data from a "video-question-answer" format to an "event span-question-answer" format, centered on events. First, high-quality SQA triples covering multiple sub-events are obtained through automatic synthesis and cleaning mechanisms. Then, pseudo-event spans are added to the original question-answering samples. Finally, a weakly supervised key event localization loss is designed using these pseudo-event spans. This allows the video question-answering model to learn the localization of question-related events while learning answer prediction, thus solving the problem of insufficient accuracy and reliability of inference results. Compared with existing technologies, this invention has at least the following beneficial effects: First: This invention expands the original sparse video question-and-answer annotation into event-centric training samples with event time spans by sampling continuous event segments and combining SQA ternaries, thereby enhancing the coverage of training data on objects, actions, temporal relationships and causal relationships.

[0019] Second: This invention achieves synthetic annotation cleaning through fragment-question-answer alignment and video text relevance ranking, which can effectively filter automatically generated samples with semantic misalignment or weak relevance, and reduce the negative impact of noisy annotations on the training of video question-answering models.

[0020] Third: This invention automatically generates pseudo-event spans for the original video question-and-answer samples, enabling the original data and synthetic data to participate in training together under a unified format of "time span-question-answer", thus making up for the lack of key event supervision in the original data.

[0021] Fourth: This invention proposes a weakly supervised key event localization loss, which guides the model to explore key event fragments related to the question without relying on precise manual boundary labeling, thereby improving the interpretability and credibility of the answer prediction basis.

[0022] Fifth: As a pluggable training enhancement framework, this invention can be used in conjunction with various end-to-end video question answering models without adding extra computing modules during the inference phase, and has good versatility and ease of deployment. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention.

[0024] Figure 2 This is a schematic diagram of the method framework of the present invention.

[0025] Figure 3 This is a schematic diagram of the SQA ternary assembly process in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the SQA ternary cleaning process in an embodiment of the present invention.

[0027] Figure 5 This is a schematic diagram of the weakly supervised key event localization training process in an embodiment of the present invention.

[0028] Figure 6 This diagram illustrates the consistency between the positioning and inference results of the present invention and existing video question answering methods TranSTR, NG+, and CRA in NExT-GQA.

[0029] Figure 7 This is a qualitative comparison chart of the inference results of the embodiments of the present invention and the existing video question answering method NG+ in NExT-GQA.

[0030] Figure 8 This is a schematic diagram of the system structure of the present invention.

[0031] Figure 9 This is a schematic diagram of the prompt template input into the video subtitle generator.

[0032] Figure 10This is a diagram illustrating the prompt template input into the question-answer synthesizer.

[0033] Figure 11 This is a schematic diagram of the prompt template input into the fragment-annotation aligner.

[0034] Figure 12 This is a schematic diagram of the prompt template input into the video timeline searcher. Detailed Implementation

[0035] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0036] Example 1 like Figure 1 As shown, this invention provides an event-centric enhancement method for video question answering, the implementation of which is as follows: S1. A time-aware description generation process is performed on the video frame sequence using a visual descriptor to obtain frame-level descriptions corresponding to each video frame. Based on these frame-level descriptions, event-centric SQA triples are generated, such as... Figure 2 As shown, its implementation method is as follows: The input video is sampled to obtain several video frames arranged in chronological order; A video-level global description is obtained by performing time-aware description generation on a video frame sequence using a visual descriptor. ; For each video frame, associate the video frame with the video-level global description. All data is input into the video descriptor to obtain the frame-level description corresponding to each video frame. Among them, frequency level global description As a time context constraint; Based on the frame-level description sequence, at least one continuous time span is randomly sampled. ,in, and These represent the start frame index and the end frame index, respectively, and satisfy the preset length constraint. , and These represent the minimum and maximum preset lengths, respectively. Extracting continuous time span Within the frame-level description, a sequence of continuous event fragment descriptions is obtained. ; Describing a sequence of consecutive event segments Input a question-answer synthesizer to generate multiple question-answer pairs that address the overall event content over a continuous time span; continuous time span By concatenating these with each question-answer pair, we obtain the event center SQA triplet.

[0037] In this embodiment, a time-aware description generation is performed on the video frame sequence based on the visual descriptor to obtain a video-level global description and a frame-level description corresponding to each video frame; continuous event segments are sampled based on the frame-level description, and the sampled continuous event segment descriptions are input into a question-answer synthesizer to generate an event-centric SQA triple containing the event time span, the synthesized question, and the synthesized answer.

[0038] In this embodiment, a video frame sequence is input into a visual descriptor to generate a video-level global description that characterizes the overall content and main events of the video. Each frame or sampled frame in the video is input into the visual descriptor in sequence, and the video-level global description is used as a temporal context constraint to generate a frame-level description that reflects the objects, actions, states, relationships and visibility attributes in the current frame.

[0039] In this embodiment, based on the constructed visual descriptor, a detailed text description with a sequential and dynamic relationship is generated for each video frame.

[0040] Specifically, for input video Perform uniform sampling or sampling at a preset frame rate to obtain A video frame arranged in chronological order. The sampling frequency can be set according to task requirements, such as 8, 16, 32, or other quantities. This invention does not limit the specific sampling method, as long as the temporal order of video events can be maintained.

[0041] Input the sampled video frame sequence into the visual descriptor ,like Figure 3 As shown, a video-level global description is obtained. A video-level global description is used to summarize the main objects, scenes, actions, and overall events in the video.

[0042]

[0043] Then, for each video frame This video frame is compared with the video-level global description. Input them into the visual descriptor to obtain the corresponding frame-level description. :

[0044] By introducing a video-level global description as context, this step enables each frame-level description to obtain cross-frame semantic constraints while preserving the visible content of the current frame, reducing inconsistencies between frame-by-frame descriptions. The frame-level description sequence is denoted as follows: This step designs different prompt templates for video-level and frame-level description generation to obtain global summaries and detailed attributes respectively, as detailed below. Figure 9 As shown. Continuous event fragment sampling and SQA triple combination: The generated time-aware description is randomly and continuously sampled to obtain an event fragment description, and a large number of diverse event-centric "event span-question-answer" triple annotations are synthesized using the constructed synthesizer. Specifically, based on frame-level description sequences Randomly sample one or more continuous time spans ,in, and These represent the start frame index and the end frame index, respectively, and satisfy the preset length constraint. , and These represent the minimum and maximum values ​​of the preset length, respectively.

[0045] Extracting time span Within the frame-level description, a sequence of continuous event fragment descriptions is obtained. Describe the sequence of events in the question-and-answer synthesizer. ,generate One question - correct answer:

[0046] in, This is used to constrain the question-answer synthesizer to generate question-answer pairs based on entire consecutive event segments, rather than generating questions based on only a single frame. The generated question-answer pairs can include descriptive, temporal, and causal question-answer pairs; wherein, the cue word template is input into the question-answer synthesizer. like Figure 10 As shown.

[0047] Time span With each question and answer pair By combining the elements, we obtain the SQA triplet at the event center. :

[0048] Repeatedly perform the continuous event fragment sampling and question-answer synthesis process to obtain a set of candidate SQA triples covering different event fragments, question types, and semantic granularities. ,in, Represents the set of events spanned in the sampling. This represents the union of all event spans.

[0049] In this embodiment, the question-answer pairs generated by the question-answer synthesizer include one or more of the following: descriptive question-answer pairs, temporal question-answer pairs, and causal question-answer pairs. Specifically, descriptive question-answer pairs are used to describe objects, scenes, actions, or attributes; temporal question-answer pairs are used to describe the process before and after an event occurs or the continuous process; and causal question-answer pairs are used to describe the cause, purpose, or result of an event.

[0050] S2. Based on event fragment-question-answer alignment and video-text relevance ranking, the event center SQA triples are cleaned to obtain a high-quality set of event center triples, such as... Figure 4 As shown, its implementation method is as follows: For candidate triplet Based on its time span, the corresponding event video segments are cropped from the input video. ;in, and These represent the start frame index and the end frame index, respectively. This indicates that the question and answer are correct. Video clips of the incident And question-answer pairs Input event snippet - Question and answer aligner, determine event video snippets Can the corresponding answer be inferred? If the result is misalignment, then the candidate triplet is discarded. If the result is alignment, then retain the candidate triplet. ; For the retained aligned triples, the event video segments are calculated using a video-text relevance sorter. Question-answer pair Cross-modal correlation scores between them; Based on cross-modal relevance scores, top-ranked videos are retained for each video. N The set of high-quality event-centric triples is obtained by finding the triples with the highest correlation. .

[0051] In this embodiment, based on the synthesized augmented triple SQA annotations, an aligner and sorter are used to filter out low-quality triple annotations that are misaligned or weakly correlated, and the remaining high-quality triple data is retained to train the model.

[0052] Specifically, since automatically generated question-answer pairs may have semantic mismatches with corresponding event fragments, this invention further incorporates an SQA triplet cleaning process. For candidate triplets... First, based on its time span, from the input video Edit the corresponding event video clips :

[0053] Video clips of the incident And question-answer pairs Input snippet - Question and answer aligner The algorithm determines whether the fragment is sufficient to infer the corresponding answer. If the result indicates misalignment, the candidate triplet is discarded; if the result indicates alignment, it is retained and proceeds to the next sorting step.

[0054] in, This represents semantically aligned video clip-question-answer triplet data; This refers to the discriminative prompt words input into the aligner, and their specific content is as follows: Figure 11 As shown.

[0055] For the retained aligned triples, a video-text relevance sorter is used. Calculate cross-modal relevance scores between event video clips and question-answer pairs, and retain top-ranked clips for each video. N The set of high-quality event-centric triples is obtained by finding the triples with the highest correlation. : .

[0056] in, Indicates video The highest similarity Video clip-question-answer triplet data.

[0057] S3. Using a video moment retrieval tool, generate corresponding pseudo-event spans for the original question and answer, and combine them with the original question and answer to form original pseudo-SQA triples; merge them with a high-quality event center triple set to construct an enhanced video question answering training set. The implementation method is as follows: Input video, original question and the original answer Enter the video timer to search for question-and-answer pairs. The most semantically relevant consecutive video segments are used to obtain the pseudo-start frame index. and pseudo-end frame index Among them, the pseudo-start frame index and pseudo-end frame index The span of pseudo-events; Connecting the span of pseudo-events with the original question and the original answer Combined, forming the original pseudo-SQA triplet. ; Let all the original pseudo-SQA triples be denoted as and the high-quality event center triplet set Merging them yields an enhanced video question-answering training set. .

[0058] In this embodiment, the original video question-and-answer dataset Typically, only the video, question, and answer are included, without the key event span used to answer the question. To enable the original question-and-answer samples to participate in training with the synthesized triples in the same training format, this invention automatically generates pseudo-event spans for each original question-and-answer sample.

[0059] Specifically, the input video Original problem and the original answer Input video time searcher Retrieve the consecutive video segments most semantically relevant to the question-answer pairs to obtain the pseudo-starting frame index. and pseudo-end frame index :

[0060] in, Indicates input to the search engine The prompt words in the text, their specific content is as follows: Figure 12 As shown.

[0061] Combining the pseudo-event span with the original question and the original answer yields the original pseudo-SQA triplet. Let all the original pseudo-SQA triples be denoted as and the high-quality event center triplet set Merging them yields an enhanced training set: .

[0062] S4. Train the video question answering model using the enhanced video question answering training set, such as... Figure 5 As shown, a weakly supervised key event localization loss is constructed using pseudo-event spans during training. This guides the video question answering model to capture key event fragments related to the question while predicting the answer, thus completing the event enhancement processing. The construction process of the weakly supervised key event localization loss includes: The multimodal fusion video representation of the video question answering model is input into the lightweight frame scoring head to obtain the frame importance score corresponding to each video frame; Based on frame importance scores, the top K video frames with the highest significance scores are selected, and the Top-ranked frames are further grouped according to the pseudo-event span of the current sample.K The frame index is divided into a set of frames within the pseudo-event span and a set of indices outside the pseudo-event span; A sampling window is constructed with the center frame of the pseudo-event span as the center, and the set of frames within the span is removed from the sampling window to obtain the set of potential positive sample frames; video frames that do not fall into the sampling window and do not belong to the set of frames within the span are identified as the set of negative sample frames. The loss for locating key events under weak supervision is calculated based on the set of potential positive sample frames and the set of negative sample frames.

[0063] In this embodiment, as Figure 5 As shown, the specific construction and enhancement process of the weakly supervised key event localization loss includes: extracting multimodal fusion video features from the video question answering model output; calculating the saliency score corresponding to each video frame using a lightweight scoring head; selecting the top K video frames with the highest saliency scores and dividing them into a frame set within the span and a frame set outside the span based on whether their time index is within the corresponding pseudo-event span; constructing a dynamic sampling window with the center of the pseudo-event span and removing video frames belonging to the frame set within the span from the dynamic sampling window to obtain a positive sample frame set; using video frames that are neither in the dynamic sampling window nor in the frame set within the span as a negative sample frame set; calculating the weakly supervised key event localization loss based on the positive and negative sample frame sets to improve the score distribution of the positive sample frame set and suppress the score distribution of the negative sample frame set; and combining the weakly supervised key event localization loss with the answer prediction loss to update the video question answering model parameters, thereby enhancing the model's feature response to key event segments.

[0064] In this embodiment, the synthesized triplet data is input into the video question answering model, and the weakly supervised temporal training loss is used to guide the model to capture key video events, thereby enhancing the model's temporal awareness capability.

[0065] Specifically, the training set will be enhanced. The video question answering model is trained using sample inputs. The model includes a visual encoder, a text encoder, a multimodal fusion module, and an answer decoder. The visual encoder and text encoder encode images and text into multimodal hidden layer representations, respectively. The multimodal fusion module bridges the multimodal semantic gap and fuses the multimodal hidden layer representations into a unified multimodal representation. The answer encoder decodes the resulting unified multimodal representation into the answer output. Let the visual encoder output be... The text encoder output is The multimodal fusion module output is ,but:

[0066] Answer decoder based on fusion representation Generate answer prediction scores using candidate answer representations and apply the video question answering answer prediction loss. Monitor and predict losses. Candidate answer cross-entropy loss, maximum margin ranking loss, or contrastive learning loss can be used, i.e.: ① Cross-entropy loss: For each containing Candidate set of elements Question-and-answer examples can be used based on video question-and-answer models. Modeling a model on candidate answers Each category distribution :

[0067] The cross-entropy at the candidate answer level is:

[0068] ② Maximum margin sorting loss: Let , interval is ,but:

[0069] ③ Contrastive learning loss: For a value of size batch and temperature parameters :

[0070] in, Indicates the first The comparison pool constructed from each video-question pair is the union of all candidate answers in the batch.

[0071] To guide the model in locating key events related to the question, this invention incorporates a lightweight frame scoring head into the video question-answering model. Multimodal fusion video representation Mapped to importance scores for each frame :

[0072] in, Indicates the first The saliency score of each video frame.

[0073] Top-ranked frames are selected based on their importance scores. K Frame Index Set And based on the pseudo-event span of the current sample Will Divided into an index set located within the pseudo-event span and the set of indices located outside the pseudo-event span .

[0074]

[0075] Since the boundaries of the pseudo-event span may be imprecise, this invention does not directly regress the start and end boundaries, but instead uses the center of the pseudo-event span. Construct a sampling window centered on And when constructing the set of potential positive sample frames, remove the top-level frames that are already within the pseudo-event span. K Frame indexing is used to avoid the model overfitting to noise boundaries.

[0076] in, Represents the set of potential positive sample frames. This represents the set of negative sample frames. Event centering enhancement and weakly supervised key event localization training can be plugged into different end-to-end video question answering models. Weakly supervised localization loss term. The calculation formula is:

[0077] in, Represents the set of potential positive sample frames. Indicates the total number of video frames. Indicates the temperature coefficient. Indicates the first One potential positive sample frame, Indicates the first One video frame, Indicates the first The significance score of each potential positive sample frame. Indicates the first The saliency score of each video frame.

[0078] Finally, the overall training objective of the video question answering model is:

[0079] in, express The weight hyperparameters are defined. Through the above training objectives, the model is explicitly guided to focus on potential keyframes near the span of pseudo-events while learning answer prediction, thereby improving its temporal awareness and the credibility of answer prediction.

[0080] In this embodiment, the overall training objective of the video question answering model includes a weighted sum of the video question answering loss and the weakly supervised localization loss; wherein, the video question answering loss is at least one of the following: candidate answer cross-entropy loss, maximum margin ranking loss, or contrastive learning loss; and the weakly supervised localization loss is... The normalized index form based on saliency score is used for calculation, and potential keyframe mining is guided by pseudo-event span, without forcing the video question answering model to regress the precise start and end boundaries of the pseudo-event span.

[0081] Experimental Results: Experiments on mainstream video question answering benchmark datasets demonstrate that the event-centric enhancement and weakly supervised temporal localization method for video question answering provided in this embodiment of the invention can effectively improve the answer prediction accuracy, key event localization capability, and inference credibility of existing video question answering models. Specifically, as follows... Figure 6 and Figure 7 As shown, the embodiments of the present invention integrate the event center enhancement module and the weakly supervised temporal localization training module with existing video question answering baseline methods such as TranSTR, NG+, and CRA, and validate them on multiple public benchmark datasets such as MSVD-QA, NExT-QA, and NExT-GQA. Experimental results show that the embodiments of the present invention achieve stable improvements in open-ended video question answering, multiple-choice video question answering, causal reasoning question answering, and video question answering tasks with key event evidence evaluation.

[0082] Table 1 shows the comparison results with the baseline method on MSVD-QA, NExT QA, and NExT-GQA. ACC@C, @T, and @D represent the sub-item accuracy for causal, temporal, and descriptive problems, respectively. @0.3 and @0.5 represent the localization performance when IoP or IoU is equal to 0.3 and 0.5, respectively, and mIoP and mIoU represent the average localization performance. ACC@GQA represents the performance when the prediction result is correct when IoP ≥ 0.5.

[0083] Table 1

[0084] Table 1 describes the test conditions and metrics: As shown in Table 1, this test was conducted on the MSVD-QA, NExT-QA, and NExT-GQA datasets. MSVD-QA was used to evaluate the overall answer prediction accuracy in open-ended video question answering tasks; NExT-QA was used to evaluate the model's ability to answer multiple-choice video questions in causal, temporal, and descriptive categories; and NExT-GQA further evaluated the answer prediction results and key event localization results. Evaluation metrics included ACC, ACC@GQA, mIoP, IoP@0.3, IoP@0.5, mIoU, IoU@0.3, and IoU@0.5. Higher values ​​for ACC metrics indicate more accurate answer predictions, while higher values ​​for IoP and IoU metrics indicate a higher degree of overlap between predicted key event segments and actual evidence segments. ACC@C, ACC@T, and ACC@D represent the sub-performance on the causal, temporal, and descriptive reasoning tasks of NExT-QA.

[0085] Combined with Table 1 and Appendix Figure 6 and attached Figure 7 The test results lead to the following conclusions: 1) Significantly Improved Overall Video Question Answering Accuracy: On the MSVD-QA dataset, after integrating the embodiments of this invention with TranSTR, the accuracy increased from 47.1% to 51.8%, achieving an absolute improvement of 4.7%; after integrating NG+, the accuracy increased from 47.3% to 50.9%, achieving an absolute improvement of 3.6%; and after integrating CRA, the accuracy increased from 46.0% to 49.5%, achieving an absolute improvement of 3.5%. These results demonstrate that the embodiments of this invention, by synthesizing SQA triples with event centers, significantly expand the coverage of video event semantics in the training data, enabling the model to more fully understand key actions, object relationships, and event content in the video, thereby improving the accuracy of the final answer prediction.

[0086] 2) Significantly Enhanced Capabilities in Complex Temporal and Causal Inference: On the NExT-QA dataset, the embodiments of this invention deliver stable improvements over all three baseline methods. Specifically, the overall accuracy of TranSTR increases from 61.5% to 66.1%, NG+ from 63.3% to 66.7%, and CRA from 63.9% to 67.2%. For causal questions, the embodiments of this invention deliver improvements of 5.9%, 4.5%, and 3.8%, respectively; for temporal questions, they deliver improvements of 4.0%, 3.2%, and 3.4%, respectively. Therefore, the embodiments of this invention do not merely improve the recognition capabilities of simple visual description questions, but rather more effectively model the temporal evolution and potential causal relationships between video events, making them particularly suitable for temporal and causal inference scenarios in complex video question answering.

[0087] 3) Simultaneous Improvement of Key Event Localization and Reliable Reasoning Capabilities: On the NExT-GQA dataset, the embodiments of this invention not only improve the accuracy of answer prediction but also significantly enhance the performance of key event localization. Taking CRA as an example, after integrating the embodiments of this invention, ACC@GQA increased from 18.2% to 20.2%, mIoP increased from 28.6% to 30.9%, and mIoU increased from 14.2% to 16.2%. Taking NG+ as an example, ACC@GQA increased from 16.0% to 18.3%, mIoP increased from 25.7% to 28.1%, and mIoU increased from 12.1% to 14.0%. The above results demonstrate that by introducing pseudo-event span and weakly supervised temporal localization loss, the embodiments of this invention enable the video question answering model not only to output correct answers but also to more accurately capture key event fragments supporting answer prediction, thereby improving the interpretability and credibility of the reasoning process.

[0088] Table 2 shows the performance comparison results of the baseline method on Causal-VidQA. ACC@D, @E, @P, and @C represent the sub-item accuracy for descriptive, explanatory, predictive, and counterfactual questions, respectively. Q→A, Q→R, and Q→AR represent the sub-item accuracy for answering only the answer, answering only the reason, and answering both, respectively.

[0089] Table 2

[0090] Table 2 describes the test conditions and metrics: As shown in Table 2, this test was conducted on the Causal-VidQA dataset. This dataset requires the model not only to predict the correct answer but also to infer its underlying causes, thus providing a more comprehensive evaluation of the model's ability to describe, explain, predict, and reason about counterfactual problems. Evaluation metrics include ACC@D, ACC@E, ACC@P, ACC@C, and ACC@ALL. ACC@P and ACC@C are further divided into Q→A, Q→R, and Q→AR, representing the accuracy of answer prediction, the accuracy of cause prediction, and the accuracy of simultaneously predicting both the answer and the cause, respectively.

[0091] Based on the test data in Table 2, the following conclusions can be drawn: 1) Significantly Improved Overall Causal Question Answering Performance: On the Causal-VidQA dataset, after integrating the embodiments of this invention with TranSTR, ACC@ALL improved from 62.2% to 68.3%, achieving an absolute improvement of 6.1%; after integrating NG+, ACC@ALL improved from 64.5% to 69.5%, achieving an absolute improvement of 5.0%; and after integrating CRA, ACC@ALL improved from 62.9% to 68.4%, achieving an absolute improvement of 5.5%. These results demonstrate that the embodiments of this invention can effectively improve the overall inference performance of existing video question answering models in causal video scenarios.

[0092] 2) Significantly Improved Answer-Cause Joint Reasoning Ability in Predictive Questions: In the Q→AR metric for predictive questions ACC@P, the implementation of this invention improved from 48.9% to 63.7% after incorporating TranSTR, an improvement of 14.8%; from 53.9% to 66.2% after incorporating NG+, an improvement of 12.3%; and from 54.6% to 63.8% after incorporating CRA, an improvement of 9.2%. This result demonstrates that the implementation of this invention can help the model simultaneously predict the correct answer and its corresponding cause, significantly reducing the risk of guessing based solely on surface relevance, and making the model more inclined to reason based on real key events in the video.

[0093] 3) Enhanced Counterfactual Reasoning Ability: In the Q→AR metrics of ACC@C counterfactual problems, the embodiments of this invention bring improvements of 4.1%, 3.2%, and 6.8%, respectively. Counterfactual problems typically require the model to understand "what would happen if a certain event did not occur or changed," demanding a high level of understanding of the causal structure of events. Experimental results show that the embodiments of this invention, through event-centric enhancement and weakly supervised temporal localization, enable the model to more fully learn the causal relationships and evidence fragments between events, thereby improving complex counterfactual reasoning ability.

[0094] Table 3 shows the performance comparison of long video complex question answering tasks. Table 3 also presents the performance comparison results with baseline methods on ActivityNet-QA.

[0095] Table 3

[0096] Table 3 describes the test conditions and metrics: As shown in Table 3, this test was conducted on the ActivityNet-QA dataset. This dataset contains longer video clips and more complex combinations of actions, scenes, and events, enabling the evaluation of the model's understanding of motion, space, temporal sequence, color, objects, location, quantity, and other categories of questions in long video scenarios. The evaluation metrics are the prediction accuracy for each category of questions and the overall accuracy ACC@ALL.

[0097] Based on the test data in Table 3, the following conclusions can be drawn: 1) Stable improvement in overall question answering performance for long videos: On the ActivityNet-QA dataset, after integrating the embodiments of this invention with TranSTR, the overall accuracy improved from 37.7% to 42.4%, an improvement of 4.7%; after integrating NG+, the overall accuracy improved from 40.2% to 44.8%, an improvement of 4.6%; and after integrating CRA, the overall accuracy improved from 42.4% to 46.8%, an improvement of 4.4%. This indicates that the embodiments of this invention can stably improve video question answering performance under conditions of long videos, multiple events, and complex scenarios.

[0098] 2) Significant improvements in motion and temporal questions: For motion-related questions, the embodiments of this invention improved TranSTR, NG+, and CRA by 9.1%, 7.1%, and 7.2%, respectively; for temporal questions, the embodiments of this invention brought improvements of 6.7%, 5.9%, and 2.3%, respectively. These results demonstrate that the event-level triples and their corresponding spans generated by the embodiments of this invention help the model capture action changes, event durations, and pre- and post-event relationships in long videos, thereby enhancing motion understanding and temporal reasoning capabilities in long video question answering.

[0099] 3) Good general applicability across multiple problem categories: In addition to motion and temporal problems, the embodiments of the present invention also achieve improvements in spatial, color, object, location, quantity, and other problem categories. For example, after integrating NG+, the embodiments of the present invention achieve a 5.3% improvement in spatial problems, a 5.1% improvement in location problems, and a 5.5% improvement in other problem categories. This indicates that the event-enhanced representation learned by the embodiments of the present invention is not limited to a single type of problem, but can achieve general event semantic understanding across multiple reasoning problems.

[0100] As shown in Table 4, Table 4 is a detailed table of offline event enhancement costs.

[0101] Table 4

[0102] Table 4 describes the test conditions and metrics: Taking the NExT-QA dataset as an example, this study statistically analyzed the memory usage, time consumption, and GPU hourly overhead of each offline processing module in the embodiments of this invention, including the visual description generator, question-answer synthesizer, fragment-annotation aligner, video text relevance sorter, and video time-of-view retrieval unit.

[0103] Based on the test data in Table 4, the following conclusions can be drawn: 1) The main computational overhead in this embodiment of the invention occurs during the offline enhancement phase: the total time for the complete offline processing flow is 18.83 hours, the total GPU hour overhead is 94.15 card-hours, and the maximum GPU memory usage is 19,290 MB. Specifically, the visual description generator uses 56.40 card-hours, the question-answer synthesizer uses 16.45 card-hours, the fragment-annotation aligner uses 19.25 card-hours, the video text relevance sorter uses only 0.10 card-hours, and the video time-of-view retrieval unit uses 1.95 card-hours. Since all of the above processing is completed offline before training, it does not increase the online inference burden after model deployment.

[0104] 2) A good balance is achieved between offline cost and training benefits: This embodiment of the invention generates and cleans event-level triples offline in one go, providing rich event semantics and temporal supervision information for subsequent training. As can be seen from the performance improvements in Tables 1 to 3, this offline processing cost can be exchanged for a 2.0% to 6.1% improvement in online inference performance on multiple video question-answering tasks, demonstrating significant data augmentation benefits.

[0105] As shown in Table 5, Table 5 is a statistical table of the number of synthetic annotations generated and retained.

[0106]

[0107] Table 5 describes the test conditions and metrics: As shown in Table 5, taking the NExT-QA dataset as an example, this embodiment of the invention first generates 156,300 candidate event-level Span-Question-Answer triples; then, it filters out semantically mismatched triples through a fragment-annotation aligner, filters out weakly related triples through a video-text relevance sorter, and finally retains high-quality triples for model training.

[0108] Based on the statistical data in Table 5, the following conclusions can be drawn: 1) This invention can generate large-scale event-level supervised data: On the NExT-QA dataset, this invention generates 156,300 candidate triples, supplementing the original training set with a large number of question-and-answer annotations built around event segments. Unlike traditional question rewriting or answer expansion, each sample generated by this invention contains the event span, question, and answer, which can simultaneously enhance semantic understanding and temporal localization supervision.

[0109] 2) The SQA cleaning mechanism effectively improves the quality of synthetic data: among the candidate triples, 32,338 were removed by the segment-annotation aligner, and 27,212 were removed by the video-text relevance sorter, ultimately retaining 96,750 high-quality triples, with a retention rate of approximately 61.9%. This result demonstrates that the embodiments of this invention do not simply add all generated samples directly to training, but rather retain more reliable event annotations through alignment verification and relevance sorting, thereby reducing the negative impact of noisy synthetic samples on model training from the source.

[0110] As shown in Table 6, Table 6 is a comparison table of inference computation amount and inference delay.

[0111] Table 6

[0112] Table 6 describes the test conditions and metrics: As shown in Table 6, this study compared the computational load, inference latency, and performance on NExT-QA of the TranSTR, NG+, and CRA models before and after integration with the embodiments of this invention. Computational load is expressed in GFLOPs, inference latency in seconds, and performance in ACC.

[0113] Based on the test data in Table 6, the following conclusions can be drawn: 1) This embodiment of the invention adds almost no online inference overhead: After integrating this embodiment, the computational cost of TranSTR changes from 158.75 GFLOPs to 160.26 GFLOPs, and the inference latency changes from 0.0623 seconds to 0.0617 seconds; the computational cost of NG+ changes from 593.42 GFLOPs to 585.83 GFLOPs, and the inference latency changes from 0.0279 seconds to 0.0282 seconds; the computational cost of CRA changes from 410.27 GFLOPs to 400.09 GFLOPs, and the inference latency changes from 0.0411 seconds to 0.0421 seconds. Therefore, this embodiment of the invention primarily improves model capabilities through data augmentation and weak supervision optimization during the training phase, without introducing significant additional computational burden during the inference phase.

[0114] 2) Significantly improves performance while maintaining basic inference efficiency: On the NExT-QA dataset, the accuracy of TranSTR improved from 61.5% to 66.1% after incorporating this embodiment of the invention, NG+ improved from 63.3% to 66.7%, and CRA improved from 63.9% to 67.2%. Therefore, this embodiment of the invention can significantly improve model inference performance while maintaining online deployment efficiency, making it suitable for resource-constrained video question-and-answer application scenarios.

[0115] As shown in Table 7, Table 7 is a comparison table of ablation experiments of key modules.

[0116] Table 7

[0117] Table 7 describes the test conditions and indicators: As shown in Table 7, this ablation experiment used NG+ as the base model and verified the effectiveness of the event center enhancement module, SQA generation module, SQA cleaning module and weakly supervised temporal localization module on the MSVD-QA, NExT-QA and NExT-GQA datasets, respectively.

[0118] Based on the test data in Table 7, the following conclusions can be drawn: 1) The event center enhancement module is indispensable: The complete method achieves 50.9% accuracy on MSVD-QA, 66.7% overall accuracy on NExT-QA, and 18.3% ACC@GQA, 28.1% mIoP, and 14.0% mIoU on NExT-GQA. After removing the event center enhancement module, the accuracy of MSVD-QA drops to 48.2%, the overall accuracy of NExT-QA drops to 63.9%, and the ACC@GQA of NExT-GQA drops to 16.8%. This indicates that relying solely on the original QA and its pseudo-span cannot fully cover the multi-event semantics in complex videos, while event center enhancement can significantly supplement the factual, temporal, and causal information in videos.

[0119] 2) Simple question rewriting cannot replace event-level enhancement: When using a syntactic enrichment strategy to replace the event-centric enhancement module in this embodiment of the invention, the accuracy of MSVD-QA is only 47.7%, the overall accuracy of NExT-QA is 63.5%, and the ACC@GQA of NExT-GQA is 16.2%. This result shows that simply rewriting the original question with synonyms can only expand the surface language form and cannot introduce new event fragments, causal relationships, and temporal supervision. Therefore, it is difficult to achieve the event-level semantic expansion effect achieved in this embodiment of the invention.

[0120] 3) The quality of the SQA generation model affects the final performance: Replacing the question-answer synthesizer in this embodiment with other generation models resulted in a decrease in overall performance. For example, after replacing it with Llama-3.1, the overall accuracy of NExT-QA dropped to 63.4%; after replacing it with GLM-4.5, the overall accuracy of NExT-QA dropped to 64.8%. This demonstrates that high-quality event-level question-answer synthesis plays a crucial role in enhancing the fine-grained semantic understanding of video question-answering models.

[0121] 4) The SQA cleaning module can suppress noisy annotations: After removing the fragment-annotation alignment step, the overall accuracy of NExT-QA decreased to 65.3%, and the mIoU of NExT-GQA decreased to 13.2%; after removing the Top-N annotation retention step, the accuracy of MSVD-QA decreased to 49.9%, and the overall accuracy of NExT-QA decreased to 64.9%. These results indicate that a certain proportion of semantically mismatched or weakly related samples exist in the generated candidate triples. A cleaning mechanism is needed to select high-quality samples in order to fully leverage the positive impact of event-augmented data on model training.

[0122] 5) Weakly supervised temporal localization loss outperforms hard boundary regression: After removing the weakly supervised localization loss, the overall accuracy of NExT-QA decreased from 66.7% to 65.7%, and the ACC@GQA of NExT-GQA decreased from 18.3% to 16.5%. When the boundary regression loss from a conventional video localization task is used to replace the weakly supervised localization loss in this embodiment of the invention, the ACC@GQA of NExT-GQA significantly decreased to 13.2%, and the mIoP decreased to 21.5%. This result shows that due to the rough boundaries and noise issues of pseudo-event spans, directly performing hard boundary regression will lead to overfitting and inaccurate boundaries. However, the weakly supervised keyframe mining strategy adopted in this embodiment of the invention can more flexibly utilize pseudo-spans and guide the model to focus on key event frames that are truly relevant to the problem.

[0123] As shown in Table 8, Table 8 is a table of direct quality inspection results for synthetic labeling.

[0124]

[0125] Table 8 describes the test conditions and indicators: As shown in Table 8, this quality inspection sampled from the synthetic annotation pool before and after cleaning, and conducted manual or model-assisted testing on the proportion of valid questions, the proportion of correct answers, the proportion of alignment span, and the overall high-quality proportion.

[0126] Based on the test data in Table 8, the following conclusions can be drawn: The quality of the synthesized annotations is significantly improved after cleaning: the original SQA annotation pool had a valid question ratio of 90.5%, a correct answer ratio of 81%, an alignment span ratio of 91.5%, and an overall high-quality ratio of 63%. After processing by the SQA cleaning module in this embodiment of the invention, the valid question ratio increases to 95%, the correct answer ratio increases to 98%, the alignment span ratio increases to 96%, and the overall high-quality ratio increases to 89%. This indicates that the SQA cleaning module can effectively filter samples with incorrect answers, misaligned events, or insufficient semantic relevance, thereby providing more reliable enhanced supervision signals for model training.

[0127] In summary, the experimental results in Tables 1 to 8 fully demonstrate that the event-centric enhancement and weakly supervised temporal localization method for video question answering provided in this invention can effectively overcome the problems of scarce training annotations, insufficient event semantic coverage, lack of supervision over key event spans, and insufficient reasoning credibility in existing video question answering methods. By automatically generating and cleaning event-level Span-Question-Answer triples, supplementing the original QA with pseudo-event spans, and adopting a weakly supervised keyframe mining training strategy, this invention can significantly improve the answer prediction accuracy, key event localization capability, complex temporal causal reasoning capability, and reasoning interpretability of the video question answering model without significantly increasing the online reasoning computation.

[0128] Example 2 like Figure 8 As shown, this invention provides an event center enhancement system for video question answering, used to execute the event center enhancement method described in Embodiment 1, including: The first processing module is used to perform time-aware description generation on the video frame sequence using a visual descriptor, obtain the frame-level description corresponding to each video frame, and generate an event-centric SQA triple based on the frame-level description. The second processing module is used to clean the event center SQA triples based on event fragment-question-answer alignment and video text relevance sorting to obtain a high-quality event center triple set. The third processing module is used to generate corresponding pseudo-event spans for the original question and answer using a video moment retrieval device, and combine them with the original question and answer to form original pseudo-SQA triples; then merge them with a high-quality event center triple set to construct an enhanced video question answering training set. The fourth processing module is used to train the video question answering model using the enhanced video question answering training set. During the training process, it constructs a weakly supervised key event localization loss using pseudo-event spans, guiding the video question answering model to capture key event fragments related to the question while predicting the answer, thus completing the enhanced processing of the event.

[0129] like Figure 8 The event center enhancement system for video question answering provided in the embodiment shown can execute the technical solution shown in the above method embodiment 1. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0130] In this embodiment, the functional units can be divided according to the event-centric enhancement method for video question answering. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this invention is illustrative and represents only a logical division; other division methods may be used in actual implementation.

[0131] In this embodiment of the invention, the event center enhancement system for video question answering, in order to achieve the principles and beneficial effects of Embodiment 1, includes hardware structures and / or software modules corresponding to the execution of various functions. Those skilled in the art should readily recognize that, in conjunction with the illustrative units and algorithm steps described in the embodiments disclosed herein, the present invention can be implemented in hardware and / or a combination of hardware and computer software. Whether a function is executed via hardware or computer software depends on the specific application and design constraints of the technical solution. Different methods can be used to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] Example 3 The present invention provides an electronic device including a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the electronic device to perform various steps of the event-centric enhancement method for video question answering as described in Embodiment 1.

[0133] In this embodiment, the electronic device may include: a processor, a memory, a bus, and a communication interface. The processor, the communication interface, and the memory are connected via the bus. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes some or all of the steps of the event center enhancement method for video question answering provided in the aforementioned embodiment 1 of this application.

[0134] Example 4 The present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the various steps of the event-centric enhancement method for video question answering as described in any of Embodiment 1.

[0135] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or combination thereof, such as static random access memory (SRAM), erasable and non-volatile read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer. The readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. The readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC), or they can exist as discrete components in a point deregistration system.

[0136] This embodiment can be provided as a method, apparatus, or computer program product; therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) including computer-usable program code. Described with reference to the flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the present invention, it should be understood that each flowchart and / or block diagram, and combinations thereof, can be implemented by computer program instructions, which can provide these computer program instructions to a computer-readable storage medium that operates in a particular manner on a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing device, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means implemented in the flowchart. Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes in the flowchart. These computer program instructions may also be accreted to a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing one or more processes and / or boxes in the flowchart. Figure 1 The steps of the function specified in one or more boxes.

[0137] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An event-centric enhancement method for video question answering, characterized in that, Includes the following steps: S1. Use a visual descriptor to perform time-aware description generation on the video frame sequence to obtain frame-level descriptions corresponding to each video frame, and generate event-centric SQA triples based on the frame-level descriptions. S2. Based on event fragment-question-answer alignment and video text relevance ranking, the event center SQA triplet is cleaned to obtain a high-quality event center triplet set. S3. Using a video moment retrieval tool, generate corresponding pseudo-event spans for the original question and answer, and combine them with the original question and answer to form the original pseudo-SQA triples; merge the set of high-quality event center triples with the original pseudo-SQA triples to construct an enhanced video question answering training set. S4. Train the video question answering model using the enhanced video question answering training set, and construct a weakly supervised key event localization loss using pseudo-event span during the training process. This guides the video question answering model to capture key event fragments related to the question while predicting the answer, thus completing the enhanced processing of the event.

2. The event-centric enhancement method for video question answering according to claim 1, characterized in that, S1 includes: The input video is sampled to obtain several video frames arranged in chronological order; A video-level global description is obtained by performing time-aware description generation on a video frame sequence using a visual descriptor. ; For each video frame, associate the video frame with the video-level global description. All data is input into the video descriptor to obtain the frame-level description corresponding to each video frame. Among them, frequency level global description As a time context constraint; Based on the frame-level description sequence, at least one continuous time span is randomly sampled. ,in, and These represent the start frame index and the end frame index, respectively, and satisfy the preset length constraint. , and These represent the minimum and maximum preset lengths, respectively. Extracting continuous time span Within the frame-level description, a sequence of continuous event fragment descriptions is obtained. ; Describing a sequence of consecutive event segments Input a question-answer synthesizer to generate multiple question-answer pairs that address the overall event content over a continuous time span; continuous time span By concatenating these with each question-answer pair, we obtain the event center SQA triplet.

3. The event-centric enhancement method for video question answering according to claim 1, characterized in that, S2 includes: For candidate triplet Based on its time span, the corresponding event video segments are cropped from the input video. ;in, and These represent the start frame index and the end frame index, respectively. This indicates that the question and answer are correct. Video clips of the incident And question-answer pairs Input event snippet - Question and answer aligner, determine event video snippets Can the corresponding answer be inferred? If the result is misalignment, then the candidate triplet is discarded. If the result is alignment, then retain the candidate triplet. ; For the retained aligned triples, the event video segments are calculated using a video-text relevance sorter. Question-Answer Pair Cross-modal correlation scores between them; Based on cross-modal relevance scores, top-ranked videos are retained for each video. N The set of high-quality event-centric triples is obtained by finding the triples with the highest correlation. .

4. The event-centric enhancement method for video question answering according to claim 1, characterized in that, S3 includes: Input video, original question and the original answer Enter the video timer to search for question-and-answer pairs. The most semantically relevant consecutive video segments are used to obtain the pseudo-start frame index. and pseudo-end frame index Among them, the pseudo-start frame index and pseudo-end frame index The span of pseudo-events; Connecting the span of pseudo-events with the original question and the original answer Combined, forming the original pseudo-SQA triplet. ; Let all the original pseudo-SQA triples be denoted as and the high-quality event center triplet set Merging them yields an enhanced video question-answering training set. .

5. The event-centric enhancement method for video question answering according to claim 1, characterized in that, The construction process of the weakly supervised critical event localization loss includes: The multimodal fusion video representation of the video question answering model is input into the lightweight frame scoring head to obtain the frame importance score corresponding to each video frame; Based on frame importance scores, the top K video frames with the highest significance scores are selected, and the Top-ranked frames are further grouped according to the pseudo-event span of the current sample. K The frame index is divided into a set of frames within the pseudo-event span and a set of indices outside the pseudo-event span; A sampling window is constructed with the center frame of the pseudo-event span as the center, and the set of frames within the span is removed from the sampling window to obtain the set of potential positive sample frames; video frames that do not fall into the sampling window and do not belong to the set of frames within the span are identified as the set of negative sample frames. The loss for locating key events under weak supervision is calculated based on the set of potential positive sample frames and the set of negative sample frames.

6. The event-centric enhancement method for video question answering according to claim 1, characterized in that, The objective function of the video question-answering model is expressed as follows: in, This represents the objective function of the video question-answering model. This represents the loss term for predicting video question-and-answer answers. This indicates the identification of loss items in key events where supervision is weak. express The weight hyperparameters, Represents the set of potential positive sample frames. Indicates the total number of video frames. Indicates the temperature coefficient. Indicates the first One potential positive sample frame, Indicates the first One video frame, Indicates the first The significance score of each potential positive sample frame. Indicates the first The saliency score of each video frame.

7. An event center enhancement system for video question answering, used to execute the event center enhancement method according to any one of claims 1-6, characterized in that, include: The first processing module is used to perform time-aware description generation on the video frame sequence using a visual descriptor, obtain the frame-level description corresponding to each video frame, and generate an event-centric SQA triple based on the frame-level description. The second processing module is used to clean the event center SQA triples based on event fragment-question-answer alignment and video text relevance sorting to obtain a high-quality event center triple set. The third processing module is used to generate corresponding pseudo-event spans for the original question and answer using a video moment retrieval device, and combine them with the original question and answer to form the original pseudo-SQA triples; the synthesized high-quality event center triples set is merged with the original pseudo-SQA triples to construct an enhanced video question answering training set; The fourth processing module is used to train the video question answering model using the enhanced video question answering training set. During the training process, it constructs a weakly supervised key event localization loss using pseudo-event spans, guiding the video question answering model to capture key event fragments related to the question while predicting the answer, thus completing the enhanced processing of the event.

8. An electronic device, characterized in that, The electronic device includes a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the electronic device to perform the steps of the event-centric enhancement method for video question answering as described in any one of claims 1-7.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the various steps of the event-centric enhancement method for video question answering as described in any one of claims 1-7.

Images