A long video understanding method based on iterative hierarchical key frame selection

Abstract

A long video understanding method based on iterative hierarchical key frame selection, first, the input video is hierarchically sampled to obtain hierarchical visual features, then a key frame query module is used to query the Nth layer visual features of the video frames and the input question text, combined with the context semantics to generate a query vector, and to predict whether it is sufficient to generate an answer to the input question, if not, update the query key frame in the higher frame rate N+1th layer visual features, and again combined with the context semantics of the input question text to iteratively query the key frame until it is sufficient to generate an answer to the question, and the query key frame is input into a multi-modal large model to generate an understanding result. The application queries the video frames based on the user input question and filters out the related frames, thereby improving the performance and efficiency of long video understanding, and has a wide application prospect in entertainment, security, education and other scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer artificial intelligence, and relates to a video understanding technology, and is a long video understanding method based on iterative hierarchical key frame selection. BACKGROUND

[0002] Long video understanding aims to enable computers to analyze the content of videos with a long duration and answer and interact according to user input questions. The length of a long video is often more than half a minute, and the length of some videos can reach several hours. In the case of limited computing resources, the extraction capability of key visual signals in the video needs to be improved to more efficiently obtain the video content. Long video understanding has a wide range of applications in the fields of entertainment, security, education and medical treatment, such as summarizing the key content of a teaching video for a lesson, which helps users quickly learn knowledge.

[0003] The technology in this field needs to extract key visual information in the video and encode it into a feature vector, and then align the visual feature vector with a large language model, so that the language model can realize multiple functions of long video understanding, such as description, question and answer, reasoning, and dialogue. In order to effectively extract long video signals related to language, researchers have proposed several methods, for example:

[0004] (1) Patent application CN110377711A “Method for solving open long video question and answer task by using hierarchical convolution self-attention network”. This method constructs a hierarchical convolution self-attention encoder network to obtain multi-layer video semantic expression for video and question, calculates a loss function for the obtained multi-layer video semantic expression of video frames, and trains a model. The hierarchical convolution self-attention network trained is used to generate answer sentences for open long video question and answer tasks. However, this scheme does not extract and select visual features for the high space-time redundancy characteristics of long videos, but encodes the entire video, and then filters the overall visual signal by combining the problem text information through the hierarchical convolution self-attention network, which has the problem of insufficient extraction of key information.

[0005] (2) Patent application CN113505680A "Content-based long-time complex scene video bad content detection method". This method uses the processed video data set for training to obtain a low-delay complex scene segmentation model; uses the low-delay complex scene segmentation model and a double-process scheduling scheme to perform video thumbnailing and key position labeling on the video; constructs an S3D-G network that separates convolution in the time and spatial domains; trains a video classification network model; and inputs the processed thumbnail video into the video classification network model for video classification of whether it contains bad information. This scheme extracts effective information in the video through the key frame selection module proposed, but the ability of this module is obtained through the learning process of the binary classification task of bad information detection, so the ability to extract problem-related information in long videos is limited, which is not conducive to downstream video understanding tasks.

[0006] (3) Patent application CN118537861A "Video semantic understanding model and generation method based on fine-grained hierarchical event features". This method uses a visual encoder to encode video frames into frame features; uses an event feature encoding layer to extract and filter event features; uses an event feature extractor to extract coupled fine-grained event features from the video through an adaptive sliding window, and uses an event feature perceiver to filter relevant event features according to user input questions. This solves the problem of coarse-grained feature extraction in existing models and reduces the number of video feature tokens. This scheme filters the event features of the input question text through the proposed event feature extractor and performs multi-layer feature filtering, but this way is a multi-layer filtering of the overall features of the video, without modeling and selecting key frames, which are important sources of information for long video understanding, and without considering the generality of the method to other long context visual signals.

[0007] (4) Paper Too Many Frames, not all Useful: Efficient Strategies for Long-Form Video QA (https: / / arxiv.org / pdf / 2406.09396). This paper adopts a three-stage approach to complete long video understanding. First, key frames are selected through unsupervised clustering of video frames and keyword matching steps. Then, a multi-modal large model is used to convert key frames into generated text descriptions. Finally, only the generated text descriptions are input into a pure language large model for reasoning. However, this method does not construct an end-to-end prediction model, i.e., there is no connection between each stage model, which cannot be adjusted based on the output feedback to better adapt to the task. At the same time, the multi-modal large model / language large model needs to perform multiple inferences in sequence during a single inference process, which is undoubtedly a huge cost in the actual production environment. SUMMARY

[0008] The technical problem to be solved by the present application is that the existing video understanding technology has two problems, one is that many methods encode the entire video and then use the designed information extraction module to filter visual information, which requires high performance and high computing resources of the video encoder and has the risk of incomplete visual information extraction; the second is that the interaction between vision and language modalities in some methods is less, and the ability to extract information related to the problem in long videos is limited, which is not conducive to downstream video understanding tasks.

[0009] The technical solution of the present application is a long video understanding method based on iterative hierarchical key frame selection, which constructs a video understanding model based on a multi-modal large model, extracts visual information step by step using hierarchical frame sampling on the input video, queries key frames in combination with input instructions or questions, and generates answers to the questions based on the understanding of the key frames, including: first, performing hierarchical frame sampling on the input video, from low frame rate sampling to high frame rate sampling, extracting hierarchical visual features from the sampling results of different frame rates by the visual encoder of the multi-modal large model, then querying the video key frames by a key frame query module based on the Nth layer visual features of the video frames and the input instruction or question text, combining the context semantics to generate a query vector, and predicting whether the query vector is sufficient to generate an answer to the input question, if not, calculating the similarity between the N+1th layer visual features with higher frame rate and the query vector of this round to obtain the N+1th layer key frame, and then iteratively querying the key frames by the key frame query module together with the input instruction or question text, until the key frames obtained by the query are sufficient to generate an answer to the question, and the multi-modal large model generates an understanding result based on the key frames obtained by the query and the instruction or question.

[0010] Further, the following steps are included:

[0011] S1, video input preprocessing and state initialization, the input of the video understanding model includes video and instructions or questions for representing understanding intention, frame sampling is performed on the entire video, frame sampling adopts a hierarchical structure, starting from low frame rate for coarse-grained sampling, and gradually to high frame rate for fine-grained sampling, obtaining multi-layer frame sampling results, and extracting visual features by the visual encoder of the multi-modal large model; at the same time, the inference state of the video understanding model is initialized, including the question to be answered or the instruction content to be executed, and the preliminary visual information composed of the currently sampled video frames;

[0012] S2. Visual language context-aware keyframe query: The keyframe query module consists of a Transformer encoder. The input is video frame visual features, instruction or question text, and a set of learnable tokens. The instruction or question and visual features are fused using the introduced learnable tokens. Then, the learnable tokens are converted into a query vector through average pooling. This vector is used to determine whether higher frame rate level visual features are needed for iterative querying. If so, the query vector is matched with higher frame rate level visual features to obtain the keyframe of the N+1th layer and add it to the query.

[0013] S3. Iterative query of keyframes: The keyframe query module uses a binary classification method to evaluate whether more granular information is needed to generate an answer based on the currently acquired query vector. If the currently extracted keyframes are insufficient to support an accurate answer to the question, the next round of query will be performed automatically. Based on the frame sampling level of a higher frame rate, keyframes at a higher frame rate will be acquired and inserted into the already acquired keyframe feature sequence in chronological order for the next round of query iteration. With each round of query iteration, the visual information required to answer the instruction or question will be gradually improved, making the query results closer to the instruction or question.

[0014] S4. After completing the keyframe extraction, summarize all the keyframes obtained from the query, arrange them in chronological order, and input them together with the input instructions or questions into the multimodal large model for reasoning. The reasoning results include answers to specific questions described in text, summaries or analyses of the video content.

[0015] The present invention implements the video understanding model through at least one instruction or at least one program. The at least one instruction or at least one program is executed by the data processor through remote or local calls, so that the video understanding model can complete the video understanding task on the input video file and instructions or questions.

[0016] This invention proposes a long video understanding method based on iterative hierarchical keyframe selection, with the following advantages: 1) Efficient information extraction: Hierarchical frame sampling is adopted, extracting visual information step by step from coarse to fine, reducing the consumption of computing resources and effectively avoiding the problem of insufficient visual information extraction. 2) Precise keyframe selection: Dynamic querying combined with user questions accurately selects the most relevant keyframes, significantly improving the accuracy of information extraction and the effect of downstream tasks. 3) Dynamic optimization: Through an iterative query mechanism, it delves into higher-level frame sampling as needed, ensuring sufficient visual details are obtained, solving the problem of limited information extraction capability under fixed query methods. 4) End-to-end processing: End-to-end multimodal large model design with tight connection between each stage improves the accuracy and efficiency of inference and reduces computational overhead.

[0017] Compared with the prior art, the present invention has the following beneficial effects.

[0018] (1) This invention enables the effective selection of key video frames in long videos. This invention constructs a hierarchical frame sampling strategy, which enables the location of key frames in long videos from coarse to fine and from the whole to the part. This process is similar to the human reasoning process, avoiding direct analysis of the entire video. This not only improves the efficiency of video analysis, but also preserves key details, thereby improving the performance and efficiency of long video understanding.

[0019] (2) This invention realizes the fusion of visual and textual information in long videos, and queries key video frames based on user input questions, making key frame selection more accurate, better understanding the semantics and visual context of user queries, and improving the accuracy of video understanding in multimodal tasks.

[0020] (3) This invention designs an iterative keyframe query self-improvement mechanism. The keyframe query module in the scheme can perform self-evaluation based on the extracted keyframe information and the accuracy of the query. By judging whether more granular frame information is needed through the decrease in perplexity, it can further improve the keyframe selection process. This design improves the method's adaptability to different lengths of user input videos. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the method of the present invention.

[0022] Figure 2 This is a schematic diagram of the video understanding model structure in the method of the present invention.

[0023] Figure 3 This is a schematic diagram of the keyframe query module preprocessing user input in the method of the present invention. Detailed Implementation

[0024] This invention proposes a long video understanding method based on iterative hierarchical keyframe selection, wherein the overall process is as follows: Figure 1 As shown, the structure of the implemented video understanding model is as follows: Figure 2As shown, the video understanding model includes a keyframe query module proposed in this invention, used to fuse visual features in the video with user-inputted text information, and combine contextual semantics to achieve accurate retrieval of video keyframes; a Multimodal Large Language Model (MLLM), which generates answers to user questions based on multimodal information understanding; and its accompanying visual encoder, used to extract visual features from video frames. In the MLLM, visual input is processed by the visual encoder, mapping it into high-dimensional visual semantic features. Commonly used visual encoders are usually based on variants of the CLIP model, such as EVA-CLIP. Then, the visual features are mapped to the embedding space of the language model through a projection layer MLP to achieve alignment between visual and linguistic features. In the language processing part, the model uses a large language model such as Vicuna to understand and generate input information using aligned visual features and text input.

[0025] The implementation of this invention is described in detail below.

[0026] Step S1: Video Input Preprocessing and State Initialization

[0027] The video input preprocessing and initialization process is as follows: Figure 3 As shown. First, the user provides a video file and related instructions or questions. The method of this invention performs frame sampling on the entire video. The frame sampling adopts a hierarchical structure, starting with coarse-grained sampling at a low frame rate and gradually progressing to fine-grained sampling at a high frame rate, for example, from frame sampling at 1 frame / second to frame sampling at 5 frames / second. Coarse-grained sampling provides a visual overview of the overall video, while high frame rate sampling captures more detailed visual information. This progressive acquisition of video content mimics the process of human reasoning about long video content: first, a general overview of the entire video is obtained, and then, based on the information needed for the query, a decision is made on whether to continue to watch in detail. Obtaining more information from the video frames helps to quickly filter and locate key frames related to the user's question in subsequent steps, thereby improving the multimodal large model's ability to analyze and understand long videos.

[0028] Simultaneously, the method of this invention initializes the inference state based on user input. This state includes: the question to be answered or the instruction to be executed, and the preliminary visual context composed of the currently sampled video frames. This initialization process lays the foundation for the subsequent dynamic iterative query operations of the keyframe query module.

[0029] Step S2: Visual Language Context-Aware Keyframe Query

[0030] After completing multi-level frame sampling of the video, this invention enters the keyframe query module. The main task of this module is to select the most relevant keyframes from the frame samples based on the user's instructions or questions. The query module consists of a Transformer encoder and introduces a set of learnable tokens. The user's input instructions or questions are combined with the already matched and filtered video frame information, and then interactively fused with the set of learnable tokens and visual language features. The learnable tokens are then converted into a query vector through average pooling. This vector is matched with the visual features of the video frames to filter keyframes. The keyframe query module of this invention employs iterative querying, starting with the visual features of the video frames corresponding to the lowest frame rate level. Based on the query vector, it determines whether higher frame rate level visual features are needed. If the query vector generated by the visual features of the video frames corresponding to the lowest frame rate level is sufficient to generate an answer, then the visual features of that level are directly used. If higher frame rate level visual features are needed, the query vector is used to match the visual features of the higher frame rate level, resulting in new keyframes added to the query. This process continues until the query vector indicates that the requirements for generating an answer are met. Then, the matched and filtered key video frames are aggregated, and the answer is generated based on the visual features of these frames. In this invention, the visual features are extracted by the visual encoder in a multimodal large model. The visual encoder, through a contrastive learning mechanism, already possesses the ability to match visual-textual information. The keyframe query module further extends this capability to include visual language context awareness.

[0031] In this stage, the query vector q∈R is calculated. d Visual features v sampled from the i-th frame of the current layer i ∈R d The keyframes of the current layer are obtained by matching the similarity between the frames. The similarity is calculated using cosine distance, as shown in the following formula:

[0032]

[0033] To achieve differentiable selection of discrete video frames during model training, this invention employs Gumbel-Softmax to generate a learnable binary index, filtering video frames in the current layer to form a subset of video frames of size P. The size of this subset varies randomly during training. Gumbel-Softmax is a sampling method from a discrete distribution; by incorporating random noise and the Softmax function, it transforms non-differentiable discrete selection into a differentiable approximation, thus allowing for optimization within the neural network. Due to the influence of random noise and similarity in Gumbel-Softmax, video frames with higher similarity have a greater probability of being selected into the subset.

[0034] During inference, the K most relevant keyframes to the current query are selected, or a threshold p is set, and frames with similarity higher than the threshold p are considered keyframes. The hyperparameters K or p for both modes are set based on task experience and video length. After each query, the keyframe query module inserts updated keyframes obtained in the current query into the existing keyframes in chronological order as input for the next iteration, ensuring that the next iteration can extract more accurate keyframes based on existing information. The layered selection of keyframes starts with coarse-grained frames and gradually progresses to fine-grained frames as needed, ensuring that inference efficiency is maintained while obtaining sufficient visual details when answering questions.

[0035] Step S3: Iteration of keyframe query

[0036] Keyframe querying is an iterative process. After each round of querying, the keyframe query module uses a binary classification method to assess whether more granular information is needed, based on the currently acquired visual context and the user question. If the extracted keyframes are insufficient to accurately answer the question, the next round of querying will be automatically initiated, delving into a higher frame rate frame sampling level to obtain more detailed visual information, i.e., keyframe features. This information is then inserted into the currently acquired keyframe feature sequence in chronological order for use in the next round of query iteration. As each round of querying progresses, the visual information required to answer the user's question is gradually improved, making the query results more closely aligned with the user's question.

[0037] For each iteration, the query vector is matched with the visual features of each frame in layer N+1 for similarity. Keyframes in layer N+1 are selected and added to the keyframes already filtered from previous layers for iterative querying. That is, in a new round of querying, the keyframe query module inputs video frames obtained by matching and filtering frame sampling information from layers 1 to N and layer N+1. Binary classification determines whether further iterative queries are needed. If the query vector satisfies the binary classification result, then all keyframes from the first layer to the current layer obtained through the current iteration matching are input into the multimodal large model for understanding and generating answers.

[0038] During training, a binary classification model is trained based on the change in perplexity of the multimodal large model's response after adding the current keyframe, deciding whether to continue keyframe querying in the current iteration. Perplexity is defined as follows:

[0039] P=2 H (p)

[0040] Where H(p) represents the cross-entropy based on the model-predicted probability distribution, and its specific form is:

[0041]

[0042] Where p(x) i ) represents the probability distribution of the true labels; q(x) i ) represents the predicted probability distribution of the model output.

[0043] The lower the perplexity, the more similar the current model's output and the true label share in reasoning logic. When the perplexity decreases, it means that more detailed visual information needs to be obtained from the next layer, and the expected output label is 1. Conversely, it means that enough information has been obtained to answer the question, and the expected output label is 0. Based on this, a binary classifier is trained to make a judgment decision.

[0044] During inference, in addition to using the trained binary classification model to determine the number of iterations, this invention sets a default maximum number of iterations for inference efficiency considerations. This number of iterations can also be set by the user. When the number of queries reaches a preset threshold, the query will end regardless of whether the currently obtained information is sufficient, and the existing query vector will be returned. This design ensures the efficiency of the method and prevents excessive computational overhead.

[0045] Step S4: Keyframe Summarization and Inference Response

[0046] After keyframe extraction, this invention aggregates all retrieved keyframes, arranges them chronologically, and inputs them along with the user's questions / instructions into a multimodal large-scale model for inference. The inference results can be presented as text descriptions, answers to specific questions, or summaries and analyses of the video content. Finally, this invention outputs the inference results to the user as an answer to the question or feedback on the execution of the instruction. This process ensures accurate and efficient responses based on multimodal information, meeting the user's query or operational needs.

[0047] Specifically, the present invention makes the following improvements.

[0048] 1) Hierarchical frame sampling strategy. By progressively increasing the frame rate from low to high, this strategy simulates the human reasoning process from the whole to the part. It first performs coarse-grained sampling, then gradually increases the fine-grained sampling based on query requirements. This hierarchical sampling not only improves the efficiency of video analysis but also preserves key details, effectively solving the problems of massive frame data volume and low reasoning efficiency in long video processing.

[0049] 2) Keyframe Query Mechanism Based on Visual Language Context Awareness. This invention combines a multimodal large-scale model's visual encoder with a text encoder to filter keyframes by calculating the similarity between the query vector and the visual features of the video frames. Building upon this, this proposal further introduces visual language context awareness, making keyframe queries more accurate and better understanding the semantics and visual context of user queries, thus improving the accuracy of video understanding in multimodal tasks.

[0050] 3) Iterative Keyframe Query Self-Improvement Mechanism. The keyframe query module in this invention has iterative optimization capabilities. That is, the system can self-evaluate based on the extracted keyframe information and the accuracy of the query, determining whether more granular frame information is needed by reducing perplexity, thereby further refining the keyframe selection. This iterative optimization mechanism improves video understanding accuracy while avoiding redundant frame processing, thus enhancing the system's inference efficiency.

[0051] This invention is based on a computer program and implements the video understanding model through at least one instruction or at least one program segment. The at least one instruction or at least one program segment is executed by a data processor via remote or local invocation to achieve the video understanding task. The instructions or program code used to implement the method of this invention can be written in any combination of one or more programming languages. The instructions or program code can be executed entirely on the data processor, partially on the processor, or as a standalone software package partially on the processor and partially on a remote device, or entirely on a remote device or server.

[0052] This invention proposes a long video understanding method based on iterative hierarchical keyframe selection. This method can query video frames based on user input questions and filter out related frames, thereby improving the performance and efficiency of long video understanding. This method has broad application prospects in entertainment, security, and education scenarios. Using this method, relevant video segments can be quickly located based on user questions, and responses can be provided, enabling a companion-style viewing experience; abnormal segments can be located in security surveillance videos with low information density, improving the efficiency of reviewing surveillance videos; and relevant content can be quickly summarized from course videos, helping to improve users' learning outcomes.

Claims

1. A long video understanding method based on iterative hierarchical key frame selection, characterized by The application discloses a video understanding model based on a multimodal large model, which gradually extracts visual information by hierarchical frame sampling on an input video, queries a key frame based on an input instruction or question, and generates an answer to the question based on understanding of the key frame, including the following steps.

2. The long video understanding method based on iterative hierarchical key frame selection according to claim 1, characterized in that The application discloses a video understanding model based on a multimodal large model, which gradually extracts visual information by hierarchical frame sampling on an input video, queries a key frame based on an input instruction or question, and generates an answer to the question based on understanding of the key frame, including the following steps. S1, video input preprocessing and state initialization, the input of the video understanding model includes a video and an instruction or a question used to represent an understanding intention, frame sampling is performed on the whole video, the frame sampling adopts a hierarchical structure, coarse-grained sampling is started from a low frame rate, and fine-grained sampling is gradually performed to a high frame rate, thereby obtaining a plurality of layers of frame sampling results, and visual features are extracted by a visual encoder of the multimodal large model; meanwhile, the inference state of the video understanding model is initialized, including a question to be answered or instruction content to be executed, and preliminary visual information composed of currently sampled video frames; S2, key frame query based on visual language context awareness, the key frame query module is composed of a Transformer encoder, the input includes video frame visual features, instruction or question text, and a group of learnable tokens, the instruction or question and the visual features are fused through the learnable tokens, then the learnable tokens are converted into a query vector through average pooling, which is used to determine whether higher frame rate hierarchical visual features are needed for iterative query, if needed, the query vector is matched with the higher frame rate hierarchical visual features to obtain the key frame of the N+1 layer for query; S3, iterative query of the key frame, the key frame query module uses a binary classification method to evaluate whether more fine-grained information is needed to generate an answer based on the currently obtained query vector, if the currently extracted key frame is not enough to support an accurate answer to the question, the next round of query is automatically performed, the key frame under a higher frame rate is obtained based on a higher frame sampling level, and the key frame is inserted into an already obtained key frame feature sequence in time sequence for query iteration in the next round, with each round of query iteration, the visual information required for answering the instruction or question is gradually improved, so that the query result is closer to the instruction or question. S4, after the key frame extraction is completed, all the key frames obtained by the query are summarized and arranged in chronological order, and the input instructions or questions are input into the multi-modal large model for reasoning. The reasoning result includes the answer to the specific question described in text, the summary or analysis of the video content.

3. The long video understanding method based on iterative hierarchical key frame selection according to claim 2, wherein in step S2, the key frame query module calculates a similarity between the visual feature v d in the current i-th frame sample of the current level and the visual feature v i ∈R d of the key frame of the current level, and the similarity is defined by a cosine distance through the following formula: During the training of the video understanding model, a Gumbel-Softmax is used to generate a learnable binary index to filter the current layer video frame and obtain a video frame subset with a size of P; During the inference of the video understanding model, the K key frames most relevant to the current query vector are directly selected by cosine distance. After each query, the key frame query module inserts and updates the key frames obtained in the current query into the existing key frames in chronological order, as the input of the next iteration of the key frame query module, so that the next iteration query is based on the existing information to extract more accurate key frames.

4. The long video understanding method based on iterative hierarchical key frame selection according to claim 2, characterized in that In step S3, the judgment of the key frame query module for query iteration is: Based on the change of the perplexity of the multi-modal large model in answering the query after the current layer key frame is added, a binary classification model is trained to decide whether to continue the key frame query in the current iteration round. The perplexity is defined as follows: P＝2 H(p) Where H(p) represents the cross-entropy based on the model prediction probability distribution, which is in the form of: where p(x i ) represents the probability distribution of the true label; q(x i ) represents the predicted probability distribution output by the model; The lower the perplexity, the more similar the output of the current model is to the true label. When the perplexity decreases, it means that more detailed visual information needs to be obtained in the next layer, and the expected output label is 1. Otherwise, it means that sufficient information has been obtained for answering, and the expected output label is 0. Based on this, a binary classifier is trained to make a decision.

5. The long video understanding method based on iterative hierarchical key frame selection according to claim 3, characterized in that When the key frame query module judges the query iteration, a default maximum iteration number is set. When the query number reaches the maximum iteration number, the query is ended regardless of whether the information obtained is sufficient, and the existing query vector is returned.

6. The long video understanding method based on iterative hierarchical key frame selection according to claim 1 or 2, characterized in that The video understanding model is implemented by at least one instruction or at least one program, which is executed by a data processor through remote calling or local calling to realize the video understanding model to complete the video understanding task for the input video file and instructions or questions.

Images