Video question and answer method for positioning key frames in overlapped grouping mode

Abstract

The invention provides a video question-answering method for positioning key frames in an overlapped grouping manner. The method comprises the following steps: firstly, extracting frame-level features and region-level features of a video, and extracting problem features; secondly, overlapping and grouping frame sequences to obtain important groups and supplementary visual information, and performing cross-modal interaction with problem features to accurately position key frames; secondly, aggregating the frame-level features and the region-level features of the key frames to obtain video aggregation features; then, according to the video aggregation feature and the problem feature, obtaining a dual-channel fusion feature; and finally, according to the fusion features and the answer features, performing answer prediction. According to the method, the interference of video redundant information on the model reasoning capability is effectively reduced, the possibility that event continuous frames are selected is improved, and meanwhile, the accuracy of video question and answer prediction is effectively improved through key region features and a double-channel fusion strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of video data processing, and in particular to a video question-answering method for overlapping grouping and positioning key frames. Background Art

[0002] With the rapid development of digital media technology, multimedia data has shown an explosive growth trend, especially video data, whose storage volume continues to rise at an alarming rate. As an important information carrier, video carries a large amount of diverse information. In-depth analysis of it has a substantial impact on the development of many industries including entertainment and medical care. Video question answering, as an emerging field of multimodal visual language research, aims to interpret the full picture of the video to answer related natural language questions. The breakthrough of this work will have a far-reaching impact on improving human-computer intelligent interaction, optimizing the level of intelligent education, and improving intelligent transportation systems.

[0003] Videos are composed of a series of continuous images. Compared with concise questions, videos often contain a lot of redundant information that is irrelevant to the questions, which may mislead the model to get the wrong answer. Therefore, accurate understanding and in-depth interpretation of video content play an important role in building an efficient video question answering model. At first, researchers mainly used the method of extracting dynamic features of videos to strengthen the key feature representation of videos. Although detailed feature extraction of videos can improve the performance of video question answering models to a certain extent, this method only weakens the interference of irrelevant information on the model, and cannot remove the redundant information problem in the video. At the same time, the information imbalance between the video and the question text also increases the complexity of feature fusion. In order to further reduce the impact of redundant information in videos, key frame selection has gradually become the focus of researchers in the field of video question answering. This method can effectively reduce the interference of redundant information in videos by selecting a small number of frames that are highly relevant to the question from the continuous frame sequence of the video and performing feature fusion.

[0004] Although keyframe screening-based methods have advanced the research on video question answering, the following problems still exist: directly locating keyframes from a large number of initially sampled frame sequences is not only difficult, but also has low positioning accuracy; secondly, the same event is often described by continuous frame fragments, and previous methods reduce the possibility of continuous frames being selected. Summary of the invention

[0005] In view of the above problems existing in the prior art, the object of the present invention is to provide a video question-answering method for overlapping grouping and positioning key frames which can avoid the above problems.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution provided by the present invention is as follows:

[0007] A video question-answering method for overlapping grouping and positioning key frames comprises the following steps:

[0008] S1. Extract frame-level feature representation and region-level feature representation for the input video; extract fine-grained feature representation of the problem corresponding to the input video content;

[0009] S2, overlappingly group the obtained frame-level features to obtain important groups of the events described by the question, and randomly select a small number of frames from relatively unimportant groups as supplementary information;

[0010] S3, cross-modal interaction is performed between the frame-level features of the acquired important video clips and supplementary information and the question features to accurately locate the key frames of the video;

[0011] S4, performing aggregation processing on the frame-level features and region-level features of the obtained video key frames to obtain video aggregation features;

[0012] S5, performing cross-modal interactive fusion on the obtained video aggregation features and question features to obtain fusion features of the video and the question;

[0013] S6. Calculate the confidence of each candidate based on the fusion features and the candidate answer features, and determine the final predicted answer based on the confidence.

[0014] Furthermore, the method of extracting the region-level features of the input video in step S1 is as follows:

[0015] Each frame of the image is divided into a fixed number of regions of equal size, and the pre-trained CLIP model is used to extract the global features of each region to obtain the region-level features of the video.

[0016] Furthermore, the specific method of obtaining important groups and supplementary information by using overlapping groups in step S2 is as follows:

[0017] First, two parameters H and L are set. H represents the number of overlapping frames after grouping, and L represents the grouping size. The specific implementation of overlapping grouping is to set a window of size L, starting from the leftmost end of the frame sequence, and move the window to the right by (LH) steps each time until it moves to the rightmost end of the frame sequence. After each movement, all frames contained in the window are divided into a group and regarded as a video segment, and finally S video segments are obtained.

[0018] The average of the frame-level features of all frames in each group is calculated to obtain the grouping features, and the important and unimportant segments of the video are determined based on the problem features.

[0019] All continuous frames in the group corresponding to the important video segment are retained, and one frame is randomly selected from each non-important group as supplementary video information.

[0020] Furthermore, the step S4 performs aggregation processing on the frame-level features and region-level features of the obtained video key frames, and the specific method is as follows:

[0021] Get the region-level features corresponding to the key frames, calculate the relevance scores of each region in each frame image and the question, select some regions with higher scores, use the Transformer decoder to perform feature aggregation, and obtain the aggregated features of the video.

[0022] Furthermore, the specific method of cross-modal interactive fusion in step S5 is as follows:

[0023] On the one hand, the aggregated features of the video are spliced ​​with the question features, and the multi-head self-attention mechanism is used to obtain the context fusion features.

[0024] On the other hand, the cross-attention mechanism is used to obtain cross-modal interactive video features based on videos and cross-modal interactive question features based on questions. The video and question features are spliced ​​to obtain symmetrical cross-modal fusion features.

[0025] Set two trainable parameters α and β to and Constraints are performed and added to obtain the fusion feature Fc.

[0026] Furthermore, the specific method of answer prediction in step S6 is as follows:

[0027] The fusion feature F c The candidate answer features are input into the Transformer decoder and classifier to obtain the probability distribution of the answer, and the one with the highest probability is selected as the final predicted answer.

[0028] The beneficial effects of adopting the above technical solution are:

[0029] The present invention provides a video question-answering method with overlapping grouped and positioned key frames, which can effectively filter redundant information in the video and capture continuous key frames that are highly relevant to the question; the present invention uses a pre-trained model to extract regional-level features of the video to enhance the video feature representation; in addition, the present invention designs a dual-channel fusion strategy to fully capture the complementary information between modalities and effectively improve the accuracy of video question-answering prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a diagram of the video question-answering model architecture for overlapping grouping and positioning key frames in an implementation of the present invention.

[0031] Figure 2 A schematic diagram of a method for overlapping grouping and cross-modal interactive positioning of key frames in an implementation of the present invention.

[0032] Figure 3 Schematic diagram of the method for local area feature aggregation in the implementation of the present invention.

[0033] Figure 4 Schematic diagram of the dual-channel feature fusion method in the implementation of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the following implementation methods are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0035] A video question answering method with overlapping grouping and positioning key frames. The model structure is as follows Figure 1 As shown in the figure, it is mainly divided into four modules: multimodal feature extraction module, key frame positioning module, regional feature aggregation module, and answer decoding module.

[0036] The above contents are explained in detail through the following steps:

[0037] S1. Extract frame-level feature representation and region-level feature representation from the video, and extract fine-grained feature representation from the problem. The specific method is as follows:

[0038] Extracting video frame-level feature representation: Use the pre-trained CLIP model to extract global features for each frame in, represents the frame-level features of the video, T represents the number of frames, and d v is the characteristic dimension of the video;

[0039] Extract regional-level feature representation of the video: Divide each frame into regions of equal size, use the pre-trained CLIP model to extract the global features of each region, and obtain the regional-level features of the video in represents the region-level features of the video, T represents the number of frames, N represents the number of regions in each frame, and d v is the characteristic dimension of the video;

[0040] Extract fine-grained feature representation of the question: Use the pre-trained DeBERTa model to extract fine-grained features of the question Among them, F q Represents the fine-grained features of the question, m represents the number of words contained in the question, and d represents the dimension of the question feature;

[0041] Since the feature dimensions of the video and the question are inconsistent, a linear visual adapter is used to unify them, and the transformed video features are obtained by the following formula:

[0042]

[0043]

[0044] Where Linear represents linear transformation, represents the transformed video frame-level features, Represents the transformed video region-level features.

[0045] S2. Use overlapping grouping to obtain important segments and supplementary information of the video. The specific method is as follows:

[0046] The initial sampling frame sequence of a video is expressed as: V = {f1, f2, f3, ..., f T}, set two parameters H and L, which represent the number of overlapping frames and the group size respectively, set the window size to L, move from left to right, move (LH) steps each time, move to the rightmost end, and get S groups. The result of overlapping grouping can be expressed as: G = {G1, G2, G3, ..., G S}={f1, f2, f3, f4), (f3, f4, f5, f6),..., (f T-3 , f T-2 , f T-1 , f T )};

[0047] The average value of the frame-level features of all frames in the i-th group is used as the feature representation of the group. The calculation formula is as follows:

[0048]

[0049] Among them, mean(·) represents the mean value calculation function, represents the frame-level features of the z-th frame of the i-th group, represents the grouping feature of the i-th group. Finally, all grouping features can be expressed as:

[0050] Use cross attention to calculate the correlation score between each grouping feature and the question feature. The calculation formula is as follows:

[0051]

[0052] in, Representative problem characteristics F q The transpose of , d represents the feature dimension, softmax represents the normalization function, G attn Represents the correlation score between the grouping feature and the problem feature;

[0053] Compare the relevance scores of each group and the question, take the group with the largest score as the important segment of the video, and retain all the frames in the group; the remaining groups are regarded as unimportant segments, and randomly retain one frame from each group, and finally obtain (S-1+L) frame images, whose frame-level features are expressed as

[0054] S3. Use cross-modal interaction to locate key frames. The specific method is as follows: perform multi-head self-attention calculation and cross-attention calculation on the frame-level features of the (S-1+L) frame image. The calculation method is as follows:

[0055]

[0056]

[0057] Among them, MSA stands for multi-head self-attention calculation method, MCA stands for multi-head cross-attention calculation method, Represents the video features further extracted by multi-head self-attention, Represents the interactive features of the video and the question, V q-attn Represents the attention score of (S-1+L) frame images and question features;

[0058] Compare the obtained attention scores and use the keyframe selector from V q-attn Select the Top-k attention scores and get the corresponding frame index to get the key frame for final positioning. The calculation method is as follows:

[0059]

[0060] Among them, Selector-Top k Represents a keyframe selector. Frame-level features representing the final localized keyframes.

[0061] S4. Aggregate the frame-level features and region-level features of the key frame. The specific method is as follows: obtain the region-level features corresponding to the key frame and express them as The multi-head self-attention mechanism is used to further extract regional features, and the cross-modal interaction mechanism is used to obtain the relevance scores of all regions in each frame with the question. The calculation formula is as follows:

[0062]

[0063]

[0064] in, Represents the regional enhanced features obtained through the multi-head self-attention mechanism, Represents the interactive features of the region and the problem, R q-attnThe attention scores representing the regional features and the problem features are obtained by using the key region selector to obtain the key region features of each frame image and aggregated with the frame-level features of the corresponding frame. The calculation formula is as follows:

[0065]

[0066]

[0067]

[0068] in, represents the selected regional features, W Q , W K , W v is the initialized trainable matrix, Selector-Top j Represents the key region selector, F v Represents the aggregated features of the video.

[0069] S5. The cross-modal interactive fusion of the video’s aggregate features and question features is mainly divided into two aspects. The specific methods are as follows:

[0070] On the one hand, context interaction fusion is performed based on the self-attention mechanism: the aggregated feature F of the video v and problem feature F q Concatenate to get the combined feature F vq , F vq Input the multi-head self-attention layer to obtain the context interaction fusion feature The calculation is as follows:

[0071] F vq =concat(F v ,F q )

[0072]

[0073] Among them, concat(·) represents the concatenation function;

[0074] On the other hand, cross-modal interaction fusion is performed based on cross attention: First, the aggregated feature F of the video is v As Q, the problem feature F q As K and V, input the cross-modal cross attention layer and the forward feedback layer to obtain the cross-modal interaction feature F guided by the video q2v , calculated as follows:

[0075]

[0076] F q2v =F qv+Linear(F q x)

[0077] Among them, F qv is the output of the cross-modal attention layer;

[0078] Secondly, the problem feature F q As Q, the aggregated features F of the video v As K and V, use F q2v The same calculation method is used to obtain the question-guided cross-modal interaction feature F v2q , splicing F q2v With F v2q Get symmetric cross-modal interactive fusion features

[0079] Set two trainable parameters α and β, both initialized to 1, to fusion features of context and cross-modal fusion features Constraints are imposed and the final fusion feature Fc of the video and the question is obtained by adding them together. The calculation method is as follows:

[0080]

[0081] S6. Calculate the confidence of each answer category based on the fusion features and candidate answer features, and determine the final predicted answer based on the confidence. The specific method is as follows:

[0082] Selective question answering: Add [CLS] tags to each answer option, use the pre-trained DeBERTa model to extract answer features, and represent the [CLS] tag features as F a Finally, F a As Q, fusion feature F c As K and V, use the Transformer-based answer decoder and classifier to obtain the probability distribution A of the candidate answers mc , calculated as follows:

[0083] A mc =MLP(Transformer_Answer_Decoder(F a , F c ))

[0084] Among them, Transformer_Answer_Decoder() represents the answer decoder, and MLP represents the answer classifier;

[0085] Select the candidate answer probability distribution A mc The one with the highest probability is taken as the predicted answer, which is calculated as follows:

[0086] Pmc = argmax(A mc )

[0087] Among them, argmax(·) means choosing the one with the highest probability as the result, P mc Indicates the predicted answer;

[0088] Open-ended question answering: A certain number of learnable tags are initialized in advance as answer queries based on the questions, and the subsequent answer prediction method is the same as that of selective question answering.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video question-answering method for overlapping grouping and positioning key frames, characterized in that: The following steps are involved: S1. For the input video, extract frame-level feature representation and region-level feature representation; for the question corresponding to the input video content, extract the fine-grained feature representation of the question; S2, overlappingly group the obtained frame-level features to obtain important groups of the events described by the question, and randomly select a small number of frames from relatively unimportant groups as supplementary information; S3, cross-modal interaction is performed between the frame-level features of the important segments of the video and the supplementary information and the problem features to accurately locate the key frames; S4, performing aggregation processing on the obtained frame-level features and region-level features of the video key frames to obtain aggregated features of the video; S5, performing dual-channel feature fusion on the obtained video aggregation features and question features to obtain fusion features of the video and the question; S6. Calculate the confidence of each candidate answer based on the fusion features and the candidate answer features, and determine the final predicted answer based on the confidence.

2. The video question-answering method of overlapping grouping and positioning key frames according to claim 1, characterized in that: The method for extracting frame-level features in step S1 is: using a pre-trained CLIP model to extract global features of each frame of the image; the method for extracting video region-level features is: dividing each frame of the image into regions of equal size, treating each region as a complete image, and using a pre-trained CLIP model to extract global features of each region to obtain region-level features of the video; the method for extracting problem features is: using a pre-trained DeBERTa model to extract fine-grained features of the problem.

3. The video question-answering method of overlapping grouping and positioning key frames according to claim 1, characterized in that: The specific method of using overlapping grouping to obtain important video clips and supplementary visual information frame-level features in step S2 is as follows: First, set the parameters H and L, which represent the number of overlapping frames and the grouping size respectively. Set a window of size L and move it from left to right, each time moving (LH) steps until it moves to the rightmost end of the frame sequence. After each movement, all frames contained in the window are divided into a group and regarded as a video clip. Afterwards, the important segments of the video are determined based on the questions. In addition, due to the differences in the complexity of each event, some events may run through multiple groups. In order to reduce the performance loss caused by missing information, for non-important groups, a frame is randomly selected as supplementary information, thereby obtaining the frame-level features of the important segments of the video and the supplementary information.

4. The video question-answering method of overlapping grouping and positioning key frames according to claim 3 is characterized in that: Based on the frame-level features of important video segments and supplementary information, cross-modal interaction is performed with question features, the relevance score is calculated and the Top-k key frames are selected.

5. The video question-answering method of overlapping grouping and positioning key frames according to claim 4, characterized in that: Based on the key frames, the corresponding region-level features are obtained, and the regional features of each frame of the image are cross-modally interacted with the problem features. The regions with high relevance to the problem are selected, and the Transformer is used to perform feature aggregation to obtain video aggregation features.

6. The video question-answering method of overlapping grouping and positioning key frames according to claim 5, characterized in that: Based on the video aggregation features and question features, the self-attention mechanism is used to obtain the contextual interaction fusion features. The cross-attention mechanism is used to obtain the video features guided by the question and the question features guided by the video and splice them to obtain the symmetric cross-modal interaction fusion features. Two trainable parameters are set to constrain and add the dual-channel fusion features.

7. The video question-answering method of overlapping grouping and positioning key frames according to claim 6, characterized in that: Based on the fusion features and question features, the Transformer decoder and answer classifier are used to obtain the probability distribution of the answer, and the one with the highest probability is selected as the final predicted answer.

Images