First-view joint video question answering and timing positioning method based on bidirectional interaction

Abstract

The application discloses a first-viewpoint joint video question-answering and timing positioning method based on bidirectional interaction, belongs to the technical field of computer vision and multi-modal information processing, and comprises the following steps: firstly, performing feature extraction on an input first-viewpoint video to obtain video timing feature representation; then, respectively constructing a time positioning branch and a question-answering branch, and realizing bidirectional information transmission between the two branches through a reciprocal interaction mechanism, so that the time positioning result provides timing constraints for the question-answering, and meanwhile, the question-answering semantic reversely guides the time positioning process; and finally, completing target time segment positioning and question answer prediction based on the features after the interaction. The application can fully mine the internal correlation between timing information and semantic information in the first-viewpoint video, improve the cooperative performance of the time positioning and the question-answering task, and has good generalization ability and application value.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal artificial intelligence, specifically to a first-person perspective joint video question answering and time-series localization method based on bidirectional interaction. Background Technology

[0002] Existing first-person perspective video understanding methods model video temporal localization and video question answering as independent tasks.

[0003] This fragmented modeling approach suffers from the following technical drawbacks: question-answering models lack precise temporal constraints and are prone to relying on prior language knowledge to generate vague or incorrect answers; temporal localization models struggle to utilize high-level semantic information, resulting in insufficient semantic meaning in the localization results; and information between the two types of tasks cannot be effectively complementary, leading to insufficient overall interpretability and robustness. Therefore, a unified method is urgently needed that enables bidirectional semantic interaction and collaborative optimization between temporal localization and question-answering tasks. Summary of the Invention

[0004] To overcome the various shortcomings caused by fragmented modeling, this invention provides a first-person perspective joint video question answering and time-series localization method based on a two-way interactive mechanism. By constructing a closed-loop structure in which localization guides question answering and the answer reinforces localization, the two tasks mutually promote each other, thereby improving the accuracy, stability and interpretability of video understanding.

[0005] To achieve the above objectives, this invention provides the following technical solution: a first-person perspective joint video question answering and time-series localization method based on a two-way interactive mechanism, comprising:

[0006] The video and the natural language question are respectively feature encoded, mapping the video to a video feature vector and the natural language question to a text feature vector;

[0007] The video features and text features are jointly modeled to obtain fused video features and fused text features;

[0008] Based on the fused video features, temporal localization processing is performed to predict video time segments related to the semantics of the natural language problem, and the corresponding time segment features are extracted.

[0009] Based on the fusion weights dynamically adjusted during the training process, the time segment features and the original video features are weighted and fused to generate fused video features for video question answering tasks.

[0010] Based on the fused video and text features, video question-and-answer answers are generated, and the credibility of the generated answers is evaluated.

[0011] When the answer meets the preset credibility condition, the answer is introduced as auxiliary text information into the time positioning process to enhance and update the time positioning result;

[0012] The temporal localization loss is calculated based on the difference between the predicted video time segments and the labeled time segments, and the video question answering loss is calculated based on the difference between the generated question answer and the real answer. The model parameters are updated by combining the two losses.

[0013] The natural language question is segmented into words, and the segmented question is mapped into a text feature vector through a text embedding layer. ,in The feature embedding dimension.

[0014] Spatiotemporal features are extracted from each video frame in the video, and the extracted features are mapped to the same feature space as the text features through a linear projection layer to obtain the video feature vector. .

[0015] In a closed-loop video question-and-answer scenario, the candidate answer text is concatenated with the natural language question and then encoded together.

[0016] Video features Text features The data are concatenated and input into a multimodal feature encoding network based on a self-attention mechanism to achieve cross-modal information interaction between video features and text features.

[0017] The fused video features and fused text features are obtained and represented as follows: ,in , .

[0018] The fused video features The input time-localization network models the video in the time dimension and predicts the video time window related to the semantics of the natural language problem. .

[0019] Based on the time window, the corresponding video time segment features are extracted from the fused video features. ,in The time segment contains the number of time steps.

[0020] Based on the current training round Total number of training rounds Calculate the fusion weight parameters that change during the training process. ,and The value range is [0,1].

[0021] Based on the fusion weight parameters, the time segment features and the original video features are weighted and summed to generate fused video features. In this study, the influence of time segment features is reduced in the early stage of training, while the role of time segment features in video question answering tasks is gradually enhanced in the later stage of training.

[0022] Integrate video features Text features Joint modeling yields the following multimodal feature representation for video question answering:

[0023] .

[0024] Based on the aforementioned multimodal feature representation, a language decoding network generates video question-and-answer answers that match the video content and the semantics of the questions. .

[0025] Based on the generation probability of each word during language decoding Calculate the perplexity of the generated answer, whereby the perplexity is expressed as... ,in The number of words used to generate the answer; when the perplexity is less than a preset threshold. If the answer is correct, it is considered a reliable answer; otherwise, it is discarded.

[0026] When the generated answer is determined to be a reliable answer, the answer is embedded and encoded to obtain the answer features. .

[0027] The video features, question features, and answer features are concatenated and input into a multimodal feature encoding network to obtain the enhanced video features, which are represented as follows:

[0028] ;

[0029] The prediction results of the time-localization network are updated based on the enhanced video features, thereby improving the accuracy of video time localization.

[0030] For video temporal localization tasks, the model needs to simultaneously identify key time intervals and regress the interval boundaries. Therefore, the temporal localization loss is designed as a combination of classification loss and regression loss, expressed as follows: .

[0031] The classification loss employs focus loss to mitigate the uneven distribution of positive and negative samples along the video timeline, making the model pay more attention to key temporal locations relevant to the question's semantics. Its definition is: .in, This represents the confidence level of the model's prediction of the target category. This is the category balance coefficient. For focusing parameters.

[0032] The regression loss uses the distance intersection-union loss to simultaneously constrain the overlap between the predicted and actual time intervals and the center position deviation. It is defined as follows: .in, Indicates the forecast time interval Compared with the actual time interval Distance from the center point This represents the minimum enclosing interval length that can simultaneously cover two intervals.

[0033] For video question answering tasks, this invention uses cross-entropy loss as the optimization objective to constrain the difference between the model-generated answers and the actual answers, expressed in the following form: .

[0034] In the joint optimization phase, the overall training objective function is constructed by weighted summing of the temporal localization loss and the question-answering loss: In a specific embodiment, to ensure the balance between the two types of tasks during training, the weight coefficients are set as follows: .

[0035] Compared to existing technologies, the advantages of this invention are as follows: This invention proposes a first-person perspective joint video question answering and temporal localization method based on bidirectional interaction to solve this problem. This invention introduces a reliable answer feedback mechanism to enhance the semantic consistency of temporal localization; it employs an adaptive dynamic feature fusion strategy during the training phase to improve model training stability; and it enhances the model's cross-modal understanding of first-person perspective videos through joint modeling based on a multimodal attention mechanism. The invention has demonstrated good experimental results on two commonly used datasets in this field: EGOTIMEQA and QAEGO4D. Attached Figure Description

[0036] Figure 1 This is a flowchart of the first-person joint video question answering and timing localization method based on two-way interaction disclosed in this example;

[0037] Figure 2 This example discloses a method for generating location-guided answers.

[0038] Figure 3 This example discloses a method for filtering answers based on perplexity.

[0039] Figure 4 shows the actual effect verification diagram disclosed in this example. Detailed Implementation

[0040] Examples will be described in detail here with reference to the accompanying drawings. The described examples are merely a part of the application scenarios of this invention and do not fully encompass the market objectives. Based on the embodiments of this invention, any embodiments implemented by other personnel without any innovative effort should fall within the scope of the claims of this invention.

[0041] Please see Figure 1 As shown, this example is a flowchart of a first-person perspective joint video question answering and time-series localization method based on two-way interaction, including:

[0042] The video and the natural language question are respectively feature encoded, mapping the video to a video feature vector and the natural language question to a text feature vector;

[0043] The video features and text features are jointly modeled to obtain fused video features and fused text features;

[0044] Based on the fused video features, temporal localization processing is performed to predict video time segments related to the semantics of the natural language problem, and the corresponding time segment features are extracted.

[0045] Based on the fusion weights dynamically adjusted during the training process, the time segment features and the original video features are weighted and fused to generate fused video features for video question answering tasks.

[0046] Based on the fused video and text features, video question-and-answer answers are generated, and the credibility of the generated answers is evaluated.

[0047] When the answer meets the preset credibility condition, the answer is introduced as auxiliary text information into the time positioning process to enhance and update the time positioning result;

[0048] The temporal localization loss is calculated based on the difference between the predicted video time segments and the labeled time segments, and the video question answering loss is calculated based on the difference between the generated question answer and the real answer. The model parameters are updated by combining the two losses.

[0049] Please see Figure 2 As shown, this example illustrates a method for generating location-guided answers, including: integrating the fused video features... The input time-localization network models the video in the time dimension and predicts the video time window related to the semantics of the natural language problem. Based on the time window, extract the corresponding video time segment features from the fused video features. ,in This refers to the number of time steps contained in the time segment. Based on the current training epoch. Total number of training rounds Calculate the fusion weight parameters that change during the training process. ,and The value range is [0,1]; based on the fusion weight parameter, the time segment features and the original video features are weighted and summed to generate fused video features. The study aims to reduce the influence of time-segment features in the early stages of training and gradually enhance their role in video question-answering tasks in the later stages. This involves fusing video features. Text features Joint modeling is performed again to obtain a multimodal feature representation for video question answering, which is represented as follows: Based on the multimodal feature representation, a language decoding network generates video question-and-answer answers that match the video content and the semantics of the questions. .

[0050] Please see Figure 3 As shown in the example, this method for filtering answers based on perplexity includes: generating video question-and-answer answers that match the video content and question semantics through a language decoding network based on the multimodal feature representation. Based on the generation probability of each word during language decoding Calculate the perplexity of the generated answer, whereby the perplexity is expressed as... ,in The number of words used to generate the answer; when the perplexity is less than a preset threshold. If the answer is correct, then the answer is considered reliable; otherwise, the answer is discarded.

[0051] Please see Figure 4 As shown, the actual effect verification diagram given in this example includes: in the first example ( Figure 4 In example A), the question is: "Where did I pick up the sieve?" This invention (corresponding to the blue part in the diagram) accurately predicts the time segment in which the action occurs and generates a semantically correct answer: "on the shelf." Although the actual labeled answer (corresponding to the green part in the diagram) states: "on the shelf," the answer generated by this invention semantically expresses the same meaning, demonstrating its ability to capture relevant contextual semantics even with slight lexical differences. In contrast, GroundVQA (corresponding to the red part in the diagram) failed to locate the correct time segment and gave an incorrect answer. This indicates that the spatiotemporally aware answer generation mechanism in this invention helps produce more faithful and interpretable prediction results in open-ended question-and-answer scenarios. In the second example ( Figure 4In scenario B), the model was asked, "What color is the fuel tank cap I closed?" and provided four candidate answers. This invention correctly selected option (B), red, and accurately located the time interval corresponding to the action of closing the fuel tank cap. While GroundVQA also selected the correct answer, it located an irrelevant video clip. This inconsistency suggests that GroundVQA may be guessing based on linguistic priors, lacking sufficient visual evidence. In contrast, the framework of this invention maintains consistency between the predicted answer and visual evidence, thereby further enhancing the model's reliability and interpretability.

[0052] This invention demonstrates its effectiveness for video question answering and time-series localization tasks through examples, as shown in Tables 1 and 2. The experimental results include three accuracy metrics: ROUGH, which measures the semantic matching between the model's predictions and the ground truth annotations; Accuracy, which represents the proportion of correctly predicted samples in the model's predictions; and IoU, which measures the overlap between the predicted and ground truth time intervals. Experiments show that this invention achieves improved accuracy compared to existing methods on the EGOTIMEQA and QAEGO4D datasets.

[0053] Table 1 shows the performance of this invention on video question answering tasks on the QAEGO4D and EGOTIMEQA datasets.

[0054] Table 1

[0055]

[0056] Table 2 shows the performance of this invention on video temporal localization tasks on the QAEGO4D and EGOTIMEQA datasets.

[0057] Table 2

[0058]

[0059] The foregoing provides the invention's content and examples, but does not cover all possible variations. Therefore, it can be understood that those skilled in the art can make certain modifications to the above content within the scope of this application.

Claims

1. A first-person perspective joint video question answering and time-series localization method based on two-way interactive interaction, characterized in that, include: Feature encoding is performed on video and natural language questions respectively, mapping video to video feature vectors and natural language questions to text feature vectors; By jointly modeling video features and text features, we can obtain fused video features and fused text features. Based on the fused video features, temporal localization processing is performed to predict video time segments related to natural language semantics and extract the corresponding time segment features; Based on the fusion weights dynamically adjusted during the training process, the time segment features and the original video features are weighted and fused to generate fused video features for video question answering tasks. Video question-and-answer answers are generated by fusing video and text features, and the credibility of the generated answers is evaluated. When the answer meets the preset credibility condition, the answer is introduced into the time positioning process as auxiliary text information; The temporal localization loss is calculated based on the difference between the predicted video time segments and the labeled time segments, and the video question answering loss is calculated based on the difference between the generated question answer and the real answer. The model parameters are updated by combining the two losses.

2. The method according to claim 1, characterized in that, Feature encoding is performed separately for video and natural language questions, mapping the video to video feature vectors and the natural language questions to text feature vectors, including: The natural language processing problem is segmented into words, and the segmented problem is mapped into a text feature vector through a text embedding layer. ,in The feature embedding dimension is used; spatiotemporal features are extracted from each video frame in the video, and the extracted features are mapped to the same feature space as the text features through a linear projection layer to obtain the video feature vector. In closed-loop video question-and-answer scenarios, candidate answer texts are concatenated with natural language questions and then encoded together.

3. The method according to claim 2, characterized in that, Joint modeling of video features and text features yields fused video features and fused text features, including: Video features Text features The features are concatenated and input into a multimodal feature encoding network based on a self-attention mechanism to obtain fused video features and fused text features, which are represented as follows: ,in , .

4. The method according to claim 3, characterized in that, Based on the fused video features, temporal localization processing is performed to predict video time segments semantically related to natural language questions, and corresponding time segment features are extracted, including: The fused video features The input time-localization network models the video in the temporal dimension and predicts video time windows that are semantically relevant to natural language questions. Based on the time window, extract the corresponding video time segment features from the fused video features. ,in This represents the number of time steps contained in a time segment.

5. The method according to claim 4, characterized in that, Based on the dynamically adjusted fusion weights during training, the temporal segment features and the original video features are weighted and fused to generate fused video features for video question answering tasks, including: Based on the current training round Total number of training rounds Calculate the fusion weight parameters that change during the training process. ,and The value range is [0,1]; based on the fusion weight parameter, the time segment features and the original video features are weighted and summed to generate fused video features. In this study, the influence of time segment features is reduced in the early stage of training, while the role of time segment features in video question answering tasks is gradually enhanced in the later stage of training.

6. The method according to claim 5, characterized in that, Video question-and-answer answers are generated based on the fusion of video and text features, and the credibility of the generated answers is evaluated, including: Integrate video features Text features Joint modeling is performed again to obtain a multimodal feature representation for video question answering, which is represented as follows: Based on multimodal feature representation, a language decoding network is used to generate video question-and-answer answers that match the video content and the semantics of the questions. Based on the generation probability of each word during language decoding Calculate the perplexity of the generated answer, where perplexity is represented as... ,in The number of words used to generate the answer; when the perplexity is less than a preset threshold. If the answer is correct, then the answer is considered reliable; otherwise, the answer is discarded.

7. The method according to claim 6, characterized in that, When the answer meets the preset credibility criteria, the answer is introduced as auxiliary text information into the time positioning process, including: When the generated answer is determined to be a reliable answer, the answer is embedded and encoded to obtain the answer features. The video features, question features, and answer features are concatenated and input into a multimodal feature encoding network to obtain the enhanced video features, which are represented as follows: Prediction results based on the enhanced video feature update time localization network.

8. The method according to claim 7, characterized in that, Temporal localization loss is calculated based on the difference between predicted video time segments and labeled time segments, and video question answering loss is calculated based on the difference between generated question-answered answers and real answers. The model parameters are updated by combining the two losses, including: The temporal localization loss is designed as a combination of classification loss and regression loss, and its form is expressed as follows: The classification loss uses focus loss, which is defined as: ,in, This represents the confidence level of the model's prediction of the target category. This is the category balance coefficient. To focus on the parameters, the regression loss is the distance intersection ratio loss, which is defined as: ,in, Indicates the forecast time interval Compared to the actual time interval Distance from the center point Let represent the minimum bounding interval length that can simultaneously cover two intervals. For video question answering tasks, cross-entropy loss is used as the optimization objective, and its form is as follows: In the joint optimization phase, the overall training objective function is constructed by weighted summing of the temporal localization loss and the question-answering loss: The weighting coefficients are set as follows: .

Images