Long video understanding method based on iterative hierarchical key frame selection

Abstract

A long video understanding method based on iterative hierarchical key frame selection comprises the following steps: firstly, carrying out hierarchical frame sampling on an input video to obtain hierarchical visual features, and then carrying out a key frame query module on the Nth-layer visual features of a video frame and an input problem text to obtain a key frame query result; video key frame query is carried out in combination with context semantics to generate query vectors, whether the query vectors are enough to generate answers to the input questions is predicted, if not, the query key frames are updated in the (N + 1) th layer of visual features with the higher frame rate, and the input question texts are combined with the context semantics to carry out iterative query of the key frames again; and inputting the inquired key frame and the instruction or the question into a multi-mode large model to generate an understanding result. According to the method, the video frames are queried and the associated frames are screened out based on the user input problem, so that the performance and efficiency of long video understanding are improved, and the method has wide application prospects in scenes such as entertainment, security and education.

Description

Technical Field

[0001] The present invention belongs to the field of computer artificial intelligence technology, relates to video understanding technology, and is a long video understanding method based on iterative hierarchical key frame selection. Background Art

[0002] Long video understanding aims to enable computers to analyze the content of videos of longer duration and to answer and interact with questions input by users. Long videos are often more than half a minute long, and some videos can be several hours long. To analyze videos of this length, with limited computing resources, it is necessary to improve the ability to extract key visual signals in the video so as to obtain video content more efficiently. Long video understanding has a wide range of applications in entertainment, security, education, and medical fields. For example, summarizing the key content of a teaching video of a course helps users learn knowledge quickly.

[0003] The technology in this field needs to extract key visual information from the video and encode it into a feature vector, and then align the visual feature vector with a large language model, so that multiple functions of long video understanding can be realized through the language model, such as description, question and answer, reasoning, dialogue, etc. In order to effectively extract long video signals related to language, researchers have proposed several methods, such as:

[0004] (1) Patent application CN110377711A "A method for solving open-ended long video question-answering tasks using a hierarchical convolutional self-attention network". This method constructs a hierarchical convolutional self-attention encoder network to obtain multi-layer video semantic expressions for videos and questions, and calculates the loss function and trains the model for the multi-layer video semantic expressions of the obtained video frames. The trained hierarchical convolutional self-attention network is used to generate answer sentences for open-ended long video question-answering tasks. However, this solution does not extract and select visual features based on the high spatiotemporal redundancy characteristics of long videos. Instead, after encoding the entire video, it uses a hierarchical convolutional self-attention network to screen the overall visual signal in combination with the question text information, which results in insufficient extraction of key information.

[0005] (2) Patent application CN113505680A "Content-based method for detecting bad content in long-duration complex scene videos". This method uses processed video data sets for training to obtain a low-latency complex scene segmentation model; uses a low-latency complex scene segmentation model and a dual-process scheduling scheme to perform video thumbnails and key position annotations on the video; constructs an S3D-G network that separates time and space domains for convolution; trains to obtain a video classification network model; and inputs the processed thumbnail video into the video classification network model to classify whether the video contains bad information. This scheme extracts valid information from the video through the key frame selection module it proposes, but the ability of this module is obtained through the learning process of the binary classification task of bad information detection. Therefore, the ability to extract problem-related information in long videos is limited, which is not conducive to downstream video understanding tasks.

[0006] (3) Patent application CN118537861A “A video semantic understanding model and generation method based on fine-grained hierarchical event features”. This method uses a visual encoder to encode video frames into frame features; uses an event feature encoding layer to extract and filter event features; uses an event feature extractor to extract coupled fine-grained event features from the video through an adaptive sliding window, and uses an event feature sensor to filter relevant event features based on the question input by the user. This solves the coarse-grained feature extraction problem existing in existing models and reduces the number of feature tokens in the video. This scheme uses the proposed event feature extractor to filter out the event features of the input question text and performs multi-layer feature filtering. However, this method is a multi-layer screening of the overall features of the video, without modeling and selecting the key frames, which are an important source of information that determines the understanding of long videos, and does not consider the generalization of the method to other long-context visual signals.

[0007] (4) Too Many Frames, not all Useful: Efficient Strategies for Long-Form Video QA (https: / / arxiv.org / pdf / 2406.09396). This paper uses a three-stage approach to complete long video understanding. First, the key frames are selected through unsupervised clustering of video frames and keyword matching steps. Then, a multimodal large model is used to convert the key frames into generated text descriptions, and finally only the obtained text descriptions are sent to the pure language large model for reasoning. However, this method does not build an end-to-end prediction model, that is, there is no connection between the models at each stage, and it cannot be adjusted according to the output feedback to better adapt to the task. At the same time, in a reasoning process, the multimodal large model / language large model needs to perform multiple reasonings in sequence, which is undoubtedly a huge cost in the actual production environment. Summary of the invention

[0008] The technical problem to be solved by the present invention is: there are two problems in the existing technology of video understanding. First, many methods encode the entire video and then filter the visual information with the help of the information extraction module designed by them. This method has high requirements on the performance and computing resources of the video encoder, and there is a risk that the visual information extraction is not comprehensive enough; second, some methods have less interaction between visual and language modalities, and the ability to extract information related to the problem in long videos is limited, which is not conducive to carrying out downstream video understanding tasks.

[0009] The technical solution of the present invention is: a long video understanding method based on iterative hierarchical key frame selection, building a video understanding model based on a multimodal large model, using hierarchical frame sampling to gradually extract visual information from the input video, combining the input instructions or questions to perform key frame queries, and generating answers to questions based on the understanding of key frames, including: firstly, hierarchical frame sampling is performed on the input video, from low frame rate sampling to high frame rate sampling, and the visual encoder of the multimodal large model extracts hierarchical visual features from the sampling results of different frame rates, and then a key frame query module performs a key frame query on the Nth layer of the video frame. The visual features and the input instructions or question text are combined with the context semantics to query the video key frames and generate a query vector. The query vector is used to predict whether it is sufficient to generate an answer to the input question. If not, the N+1th layer of visual features with a higher frame rate is used to calculate the similarity with the current query vector to obtain the N+1th layer of key frames. Together with the key frames of the first N layers that have been obtained, the key frames are iteratively queried together with the input instructions or question text by the key frame query module until it is sufficient to generate an answer to the question. The key frames obtained by the query and the instructions or questions are combined by the multimodal large model to generate an understanding result.

[0010] Further, the following steps are included:

[0011] S1. Video input preprocessing and state initialization. The input of the video understanding model includes video and instructions or questions used to express the understanding intention. The entire video is frame sampled. The frame sampling adopts a hierarchical structure, starting from coarse-grained sampling at low frame rate and gradually moving to fine-grained sampling at high frame rate to obtain multi-layer frame sampling results. The visual encoder of the multimodal large model extracts visual features. At the same time, the reasoning state of the video understanding model is initialized, including the question to be answered or the content of the instruction to be executed, as well as the preliminary visual information composed of the currently sampled video frames.

[0012] S2. Key frame query with visual language context awareness. The key frame query module is composed of a Transformer encoder. Its input is the visual features of the video frame, the instruction or question text, and a set of learnable tokens. The instruction or question is fused with the visual features through the introduced learnable tokens. Then, the learnable tokens are converted into a query vector through average pooling to determine whether visual features at a higher frame rate level are needed for iterative query. If necessary, the query vector is matched with the visual features at a higher frame rate level to obtain the key frame at the N+1th layer and add it to the query.

[0013] S3. Iterative query of key frames. The key frame query module uses a binary classification method to evaluate whether more fine-grained information is needed to generate an answer based on the currently acquired query vector. If the currently extracted key frames are not sufficient to support accurate answers to questions, the next round of queries will be automatically performed. Based on the frame sampling level with a higher frame rate, key frames at a higher frame rate will be acquired and inserted into the updated key frame feature sequence in chronological order for the next round of query iterations. With each round of query iterations, the visual information required to answer instructions or questions is gradually improved, making the query results closer to instructions or questions.

[0014] S4. After completing the key frame extraction, all the key frames obtained from the query are summarized and arranged in chronological order. They are input into the multimodal large model together with the input instructions or questions for reasoning. The reasoning results include answers to specific questions described in text, and a summary or analysis of the video content.

[0015] The present invention implements the video understanding model through at least one instruction or at least one program. The at least one instruction or at least one program is used to be executed by a data processor through remote call or local call, so as to implement the video understanding model to complete the video understanding task for the input video file and instructions or questions.

[0016] The present invention proposes a long video understanding method based on iterative hierarchical key frame selection, which has the following advantages: 1) Efficient information extraction: hierarchical frame sampling is used to gradually extract visual information from coarse to fine, which reduces the consumption of computing resources and effectively avoids the problem of insufficient visual information extraction. 2) Accurate key frame selection: Dynamic queries are performed in combination with user questions to accurately select the most relevant key frames, which significantly improves the accuracy of information extraction and the effectiveness of downstream tasks. 3) Dynamic optimization: Through an iterative query mechanism, deeper into higher-level frame sampling is performed as needed to ensure that sufficient visual details are obtained, solving the problem of limited information extraction capabilities under fixed query methods. 4) End-to-end processing: End-to-end multimodal large model design, with each stage closely connected, improves the accuracy and efficiency of reasoning and reduces computational overhead.

[0017] Compared with the prior art, the present invention has the following beneficial effects.

[0018] (1) The present invention realizes the effective selection of key video frames in long videos. The present invention constructs a hierarchical frame sampling strategy to realize the positioning of key frames in long videos from coarse to fine and from overall to local. This process is similar to the process of human reasoning and avoids directly analyzing the entire video. It not only improves the efficiency of video analysis, but also retains key details, improving the performance and efficiency of long video understanding.

[0019] (2) The present invention realizes the fusion of visual and textual information in long videos, and queries key video frames based on questions input by users, making key frame selection more accurate, and can better understand the semantics and visual context association of user queries, thereby improving the accuracy of video understanding in multimodal tasks.

[0020] (3) The present invention designs a self-improvement mechanism for iterative key frame query. The key frame query module in the scheme can perform self-evaluation based on the extracted key frame information and the accuracy of the query, and judge whether more fine-grained frame information is needed by reducing the perplexity, so as to further improve the key frame selection process. This design improves the method's ability to adapt to different length videos input by users. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the process of the present invention.

[0022] Figure 2 Schematic diagram of the video understanding model structure in the method of the present invention.

[0023] Figure 3 This is a schematic diagram of preprocessing of user input by a key frame query module in the method of the present invention. DETAILED DESCRIPTION

[0024] The present invention proposes a long video understanding method based on iterative hierarchical key frame selection, wherein the overall process is as follows: Figure 1 As shown, the structure of the implemented video understanding model is as follows Figure 2As shown, the video understanding model includes the key frame query module proposed by the present invention, which is used to fuse the visual features in the video with the text information input by the user, and combine the context semantics to realize the accurate retrieval of the video key frames, the multimodal large language model (MLLM), which generates answers to user questions based on the understanding of multimodal information, and its supporting visual encoder, which is used to extract the visual features of the video frame. In the multimodal large model, the visual input is processed by the visual encoder and mapped into high-dimensional visual semantic features. Commonly used visual encoders are usually based on variants of the CLIP model, such as EVA-CLIP. Then, the visual features are mapped to the embedding space of the language model through a projection layer MLP to achieve the alignment of visual and language features. In the language processing part, the model uses a large language model such as Vicuna, and completes the understanding and generation of the input information through the aligned visual features and text input.

[0025] The implementation of the present invention is described in detail below.

[0026] Step S1: Video input preprocessing and state initialization

[0027] The video input preprocessing and initialization process is as follows Figure 3 As shown. First, the user provides a video file and related instructions or questions. The method of the present invention performs frame sampling on the entire video. The frame sampling adopts a hierarchical structure, starting from a low frame rate for coarse-grained sampling, and gradually to a high frame rate for fine-grained sampling, for example, from 1 frame / second frame sampling, and gradually to 5 frames / second frame sampling. Coarse-grained sampling provides a visual overview of the video as a whole, while high-frame rate sampling captures more detailed visual information. This layer-by-layer progressive acquisition of video content imitates the process of humans gradually reasoning about the content of long videos: first, make a general overview of the entire video, and then decide whether to continue to watch it in detail based on the information that needs to be queried, and obtain more information from the video frames, which helps to quickly screen and locate key frames related to user questions in subsequent steps, thereby improving the ability of multimodal large models to analyze and understand long videos.

[0028] At the same time, the method of the present invention initializes the reasoning state according to the user's input. This state includes: the question to be answered or the content of the instruction to be executed, and the preliminary visual context composed of the currently sampled video frames. This initialization process lays the foundation for the subsequent dynamic iterative query operation of the key frame query module.

[0029] Step S2: Vision-language context-aware keyframe query

[0030] After completing the multi-level frame sampling of the video, the present invention enters the key frame query module, the main task of which is to select the most relevant key frame from the frame sampling according to the user's instructions or questions. The query module is composed of a Transformer encoder and introduces a set of learnable tokens. The instructions or questions input by the user are combined with the video frame information that has been matched and screened, and the interactive feature fusion is performed with the set of learnable tokens and visual language features. The learnable tokens are converted into a query vector through average pooling. This vector is matched with the visual features of the video frame for filtering key frames. The key frame query module of the present invention adopts iterative query, starting from the visual features of the video frame corresponding to the layer with the lowest frame rate, and judging whether the visual features of the higher frame rate layer are needed according to the query vector. If the query vector generated by the visual features of the video frame corresponding to the layer with the lowest frame rate is sufficient to generate an answer, the visual features of this layer are directly used. If the visual features of a higher frame rate layer are needed, the query vector is used to match the visual features of the higher frame rate layer to obtain a new key frame to add to the query until the query vector indicates that the need to generate an answer is met. The key video frames that have been matched and screened are summarized, and the answers are generated by understanding the visual features of these frames. In the present invention, the visual features are extracted by the visual encoder in the multimodal large model. The visual encoder has the ability to match visual-text information through a comparative learning mechanism. On this basis, the key frame query module is further expanded to have the function of visual language context perception.

[0031] At this stage, by calculating the query vector q∈R d and the visual feature v in the i-th frame sample of the current level i ∈R d The key frame of the current layer is obtained by matching the similarity between them. The similarity adopts the cosine distance, and the formula is as follows:

[0032]

[0033] During model training, in order to achieve differentiable discrete video frame screening and selection, the present invention uses Gumbel-Softmax to generate a learnable binary index to screen the current layer video frames to form a video frame subset of size P. The size of the subset changes randomly during training. Gumbel-Softmax is a method for sampling from a discrete distribution. By adding random noise and Softmax function, non-differentiable discrete selection is converted into a differentiable approximation, which can be optimized in a neural network. Affected by the random noise and similarity size of Gumbel-Softmax, the higher the similarity, the greater the probability that the video frame is selected into the subset.

[0034] During reasoning, select the K key frames most relevant to the current query or set a threshold p, and select the key frames with a similarity higher than the threshold p as key frames. The hyperparameters K or p of the two modes are set according to task experience and video length. After each query, the key frame query module will insert and update the key frames obtained in this round of query in the existing key frames in chronological order as the input of the next iteration of the query module, ensuring that the next round of iterative queries can perform more accurate key frame extraction based on the existing information. The layer-by-layer selection of key frames starts from coarse-grained frames and gradually advances to fine-grained frames as needed to ensure that both reasoning efficiency and sufficient visual details can be maintained when answering questions.

[0035] Step S3: Iteration of key frame query

[0036] Keyframe query is an iterative process. After each round of query, the keyframe query module will evaluate whether more fine-grained information is needed based on the currently acquired visual context and user questions through a binary classification method. If the currently extracted keyframes are not sufficient to support accurate answers to questions, the next round of query will be automatically performed, going deeper into the frame sampling level with a higher frame rate to obtain more detailed visual information, namely keyframe features, and inserting them in chronological order to update the currently acquired keyframe feature sequence for the next round of query iterations. As each round of query deepens, the visual information required to answer user command questions is gradually improved, making the query results closer to user questions.

[0037] For each round of iteration, the query vector is matched with the visual features of each frame of the N+1th layer for similarity, and the key frames of the N+1th layer are selected and added to the key frames that have been selected in the previous layer for iterative query. That is, in a new round of query, the key frame query module input is the video frame obtained by matching and screening based on the frame sampling information of the 1st to Nth layers and the N+1th layer. The binary classification is responsible for deciding whether further iterative query is needed. When the query vector satisfies the binary classification result, the key frames of all frames matched in the current iteration, that is, all key frames obtained from the first layer to the current layer, are input into the multimodal large model for understanding and generating answers.

[0038] During the training process, the binary classification model is trained based on the change in perplexity of the multimodal large model after adding the current layer key frame, and the decision is made whether to continue the key frame query in the current iteration round. The perplexity is defined as follows:

[0039] P = 2 H (p)

[0040] Among them, H(p) represents the cross entropy based on the model prediction probability distribution, and its specific form is:

[0041]

[0042] Where p(x i ) represents the probability distribution of the true label; q(x i ) represents the predicted probability distribution of the model output.

[0043] The lower the perplexity, the more similar the reasoning logic is between the output of the current model and the true label. When the perplexity decreases, it means that it is necessary to further obtain more detailed visual information of the next layer, and the expected output label is 1; otherwise, it means that enough information has been obtained to answer, and the expected output label is 0. Based on this, a binary classifier is trained to make judgment decisions.

[0044] During reasoning, in addition to using the binary classification model obtained through training to determine the number of iterations, the present invention sets a default maximum number of iterations for the sake of reasoning efficiency, which can also be set by the user. When the number of queries reaches a preset threshold, the query will be terminated regardless of whether the currently acquired information is sufficient, and the existing query vector will be returned. This design ensures the efficiency of the method and prevents excessive computational overhead.

[0045] Step S4: Key frame aggregation and inference response

[0046] After completing the key frame extraction, the present invention summarizes all the key frames obtained by the query, arranges them in chronological order, and inputs them into the multimodal large model together with the user's questions / instructions for reasoning. The reasoning results can be presented in the form of text descriptions, answers to specific questions, or summaries and analysis of video content. Finally, the present invention outputs the reasoning results to the user as an answer to the question or feedback on the execution of the instruction. This process ensures that accurate and efficient responses can be provided based on multimodal information to meet the user's query or operation needs.

[0047] Specifically, the present invention makes the following improvements.

[0048] 1) Hierarchical frame sampling strategy. By gradually increasing from low frame rate to high frame rate, the reasoning process of humans from the whole to the part is simulated. Coarse-grained sampling is performed first, and then the fine-grained sampling is gradually increased according to the query requirements. This hierarchical sampling not only improves the efficiency of video analysis, but also retains key details, effectively solving the problem of large frame data volume and low reasoning efficiency in the process of long video processing.

[0049] 2) Key frame query mechanism based on visual language context perception. The present invention combines the visual encoder of the multimodal large model with the text encoder to select key frames by calculating the similarity between the query vector and the visual features of the video frame. On this basis, this proposal further introduces the visual language context perception capability, making key frame queries more accurate, and being able to better understand the semantics of user queries and the visual context association, thereby improving the accuracy of video understanding in multimodal tasks.

[0050] 3) Self-improvement mechanism of iterative key frame query. The key frame query module in the present invention has the ability of iterative optimization, that is, the system can self-evaluate based on the extracted key frame information and the accuracy of the query, and judge whether more fine-grained frame information is needed by the decrease in perplexity to further improve the key frame selection. This iterative optimization mechanism improves the accuracy of video understanding while avoiding redundant frame processing and improving the reasoning efficiency of the system.

[0051] The present invention is implemented based on a computer program, and the video understanding model is implemented by at least one instruction or at least one program, and the at least one instruction or at least one program is used to be executed by a data processor through a remote call or a local call to implement the video understanding task. The instructions or program codes used to implement the method of the present invention can be written in any combination of one or more programming languages. The instructions or program codes can be executed entirely on a data processor, partially on a processor, partially on a processor as an independent software package and partially on a remote device, or entirely on a remote device or server.

[0052] The present invention proposes a long video understanding method based on iterative hierarchical key frame selection, which can query video frames based on user input questions and filter out related frames, thereby improving the performance and efficiency of long video understanding. The method of the present invention has broad application prospects in entertainment, security, education and other scenarios. Through this method, relevant video clips can be quickly located according to the user's questions, and replies can be given to achieve companion viewing; abnormal clips can be located in security scene monitoring videos with low information density, improving the efficiency of reviewing monitoring videos; relevant content can also be quickly summarized for course videos to assist in improving users' learning effects on the course.

Claims

1. A long video understanding method based on iterative hierarchical key frame selection, characterized by A video understanding model is constructed based on a multimodal large model, and visual information is gradually extracted from the input video using hierarchical frame sampling. Key frame queries are performed in combination with input instructions or questions, and answers to questions are generated based on the understanding of the key frames, including: first, hierarchical frame sampling is performed on the input video, from low frame rate sampling to high frame rate sampling, and the visual encoder of the multimodal large model extracts hierarchical visual features from the sampling results of different frame rates. Then, a key frame query module queries the video key frames based on the Nth layer of visual features of the video frame and the input instruction or question text in combination with context semantics to generate a query vector. Whether the query vector is sufficient to generate an answer to the input question is predicted, if not, the N+1th layer of visual features with a higher frame rate is used to calculate the similarity with the query vector of this round to obtain the N+1th layer of key frames, and the key frames of the first N layers that have been obtained are again iterated by the key frame query module with the input instruction or question text until it is sufficient to generate an answer to the question, and the key frames obtained by the query and the instruction or question are combined by the multimodal large model to generate an understanding result.

2. A long video understanding method based on iterative hierarchical key frame selection according to claim 1, characterized in that The following steps are involved: S1. Video input preprocessing and state initialization. The input of the video understanding model includes video and instructions or questions used to express the understanding intention. The entire video is frame sampled. The frame sampling adopts a hierarchical structure, starting from coarse-grained sampling at low frame rate and gradually moving to fine-grained sampling at high frame rate to obtain multi-layer frame sampling results. The visual encoder of the multimodal large model extracts visual features. At the same time, the reasoning state of the video understanding model is initialized, including the question to be answered or the content of the instruction to be executed, as well as the preliminary visual information composed of the currently sampled video frames. S2. Key frame query with visual language context awareness. The key frame query module is composed of a Transformer encoder. Its input is the visual features of the video frame, the instruction or question text, and a set of learnable tokens. The instruction or question is fused with the visual features through the introduced learnable tokens. Then, the learnable tokens are converted into a query vector through average pooling to determine whether visual features at a higher frame rate level are needed for iterative query. If necessary, the query vector is matched with the visual features at a higher frame rate level to obtain the key frame at the N+1th layer and add it to the query. S3. Iterative query of key frames. The key frame query module uses a binary classification method to evaluate whether more fine-grained information is needed to generate an answer based on the currently acquired query vector. If the currently extracted key frames are not sufficient to support accurate answers to questions, the next round of queries will be automatically performed. Based on the frame sampling level with a higher frame rate, key frames at a higher frame rate will be acquired and inserted into the updated key frame feature sequence in chronological order for the next round of query iterations. With each round of query iterations, the visual information required to answer instructions or questions is gradually improved, making the query results closer to instructions or questions. S4. After completing the key frame extraction, all the key frames obtained from the query are summarized and arranged in chronological order. They are input into the multimodal large model together with the input instructions or questions for reasoning. The reasoning results include answers to specific questions described in text, and a summary or analysis of the video content.

3. The long video understanding method based on iterative hierarchical key frame selection according to claim 2 is characterized in that in step S2, the key frame query module calculates the query vector q∈R d and the visual feature v in the i-th frame sample of the current level i ∈R d The key frame of the current layer is obtained by matching the similarity between them. The similarity is defined by the cosine distance using the following formula: When training the video understanding model, Gumbel-Softmax is used to generate a learnable binary index to filter the current layer video frames to obtain a video frame subset of size P; When the video understanding model is inferring, the K key frames most relevant to the current query vector are directly screened out through the cosine distance. After each query, the key frame query module inserts the existing key frames in chronological order to update the key frames obtained in this round of query, which serve as the input of the next iteration of the key frame query module, so that the next round of iterative query can perform more accurate key frame extraction based on the existing information.

4. The long video understanding method based on iterative hierarchical key frame selection according to claim 2, characterized in that In step S3, the key frame query module determines the query iteration as follows: Based on the change in perplexity of the multimodal large model answer after adding the current layer key frame, the binary classification model is trained to decide whether to continue the key frame query in the current iteration round. The perplexity is defined as follows: P＝2 H(p) Among them, H(p) represents the cross entropy based on the model prediction probability distribution, which is in the form of: Where p(x i ) represents the probability distribution of the true label; q(x i ) represents the predicted probability distribution of the model output; The lower the perplexity, the more similar the reasoning logic is between the output of the current model and the true label. When the perplexity decreases, it means that it is necessary to further obtain more detailed visual information of the next layer, and the expected output label is 1; otherwise, it means that enough information has been obtained to answer, and the expected output label is 0. Based on this, a binary classifier is trained to make judgment decisions.

5. The long video understanding method based on iterative hierarchical key frame selection according to claim 3 is characterized by: When the key frame query module determines the query iteration, it sets a default maximum number of iterations. When the query number reaches the maximum number of iterations, the query is terminated regardless of whether the currently acquired information is sufficient, and the existing query vector is returned.

6. A long video understanding method based on iterative hierarchical key frame selection according to claim 1 or 2, characterized in that The video understanding model is implemented by at least one instruction or at least one program, and the at least one instruction or at least one program is used to be executed by a data processor through remote call or local call, so that the video understanding model completes the video understanding task for the input video file and instructions or questions.

Images