A sampling method and system for long video understanding

Abstract

The present invention relates to the field of computer vision, and specifically to a sampling method and system for long video understanding, comprising the following steps: obtaining a long video and video subtitles as input, dividing the long video into multiple semantically consistent video segments using a large video model; mapping each video segment to obtain an input sequence through a visual encoder, inputting the input sequence into a large video language model, and generating a text summary of the video segment; performing token sampling on the video segment, calculating the relative weights between the frames of the video segment, and assigning a different number of tokens to each frame based on the weights; inputting the token-sampled video frames and the text summary into the large video language model, and obtaining the key frames of the video segment based on the matching degree between the video frame tokens and the text summary. Compared with the prior art, the present invention utilizes a token sampling method to improve frame processing efficiency, solving the problem of the difficulty in balancing the memory overhead of the multimodal large model and the integrity of the video information in the existing long video understanding methods.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an efficient sampling method and system for long video understanding. Background Art

[0002] In recent years, with the rapid development of information technology and the continuous improvement of network infrastructure, global video data has seen explosive growth. These videos cover a wide range of scenarios, from daily life records to professional applications. Faced with this massive and diverse video data, how to effectively understand and extract key information has become a research focus in the field of video understanding. Intelligent video understanding technology is a cutting-edge technology that has rapidly developed in this context and has attracted widespread attention from academia and industry. By leveraging the latest research results in deep learning, natural language processing, and computer vision, this technology is dedicated to automatically analyzing complex video content, extracting key information, and performing structured annotation and summarization of video content, greatly improving the understandability and usability of video data.

[0003] The Large Video Language Model (LLM) is one of the latest breakthroughs in video understanding. By introducing a large-scale pre-trained model and combining video understanding technology with a large language model, the LLM is able to generate text based on video content, achieving both semantic understanding of video content and natural language generation. This technology not only improves the intelligence of video understanding but also demonstrates excellent performance in tasks such as video question-answering, video summary generation, and multimodal interaction. However, despite its widespread application and significant success in video understanding, the LLM still faces the following challenges when it comes to understanding long videos: 1) insufficient video temporal segmentation capabilities; 2) inaccurate keyframe selection; and 3) high computational resource consumption.

[0004] Video temporal segmentation is a key step in understanding long videos. Leveraging the powerful semantic analysis capabilities of large video language models, temporal segmentation can divide long videos into multiple semantically independent segments based on content themes, laying the foundation for subsequent understanding. Video temporal segmentation aims to divide videos into semantically coherent and independent segments. Compared to the original long video, these segments contain fewer frames and are more suitable for further processing. Existing methods typically combine scene detection tools with similarity calculations or employ specialized time series labeling techniques for temporal localization. Most existing models use fixed time intervals or scene-based segmentation for temporal segmentation. While these methods are effective for short videos or videos with relatively simple content structures, in long videos, fixed time intervals or scene-based segmentation methods struggle to accurately identify semantic boundaries between segments, as they often arise from topic transitions or content changes. This results in segmented segments lacking coherence and logical logic. Furthermore, semantic transitions in videos may not always sync with changes in the visual scene. Relying solely on visual features for temporal segmentation often results in discontinuous semantic information within the segmented segments.

[0005] After completing the temporal segmentation of the video, each video clip may still contain a large amount of redundant information, so it is necessary to further screen out the most representative key frames to condense the information of the clip. Key frame selection is the core link in generating high-quality video understanding, which directly determines the simplicity of video content presentation and its information carrying capacity. Many studies have been devoted to reducing the number of input frames required for downstream tasks such as action recognition, video temporal localization, and video question answering. These methods include compressing frames by calculating inter-frame similarity, utilizing temporal attention mechanisms, and retrieving key frames through large language model outputs. Existing methods usually adopt a fixed-step frame sampling strategy or select key frames based on visual similarity, but this method ignores the differences in information density between frames, which may result in missing key frames in clips with concentrated key information, or collecting a large number of redundant frames in clips with sparse information.

[0006] In addition to the aforementioned limitations, existing models consume significant computational resources when completing long video understanding tasks. A notable characteristic of long videos is their length and the sheer volume of information they contain. Therefore, models must process a large number of frames, leading to a sharp increase in video memory consumption and computational complexity. For example, at one frame per second, a 5-minute video will generate 300 frames. If each frame generates approximately 700 visual tokens, the total number of tokens will exceed 200,000, exceeding the processing capacity of most models. Summary of the Invention

[0007] The purpose of the present invention is to overcome the problems of the above-mentioned existing long video understanding methods, such as poor performance in video timing segmentation and key frame selection tasks and consumption of a large amount of computing resources, and to provide a sampling method for long video understanding.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] As a first aspect of the present invention, a sampling method for long video understanding is provided, comprising the following steps:

[0010] Take a long video and its subtitles as input, and use the video model to split the long video into multiple semantically consistent video segments;

[0011] For each video clip, the visual encoder is used to map the input sequence, which is then fed into the video language model to generate a text summary of the video clip.

[0012] Perform token sampling on the video clip, calculate the relative weights between the frames of the video clip, and assign different numbers of tokens to each frame according to the weights;

[0013] The video frames and text summaries after token sampling are input into the video language model, and the key frames of the video clips are obtained based on the matching degree between the video frame tokens and the text summary.

[0014] As a preferred technical solution, the process of obtaining the video clip is as follows:

[0015] The text sequence of the input video subtitles is processed by the word segmenter and split into basic language units;

[0016] The text sequence passes through the embedding layer, mapping each language unit to a corresponding vector representation;

[0017] The vector representation undergoes dimension transformation through the projection layer and is adjusted to the input dimension of the video large language model to obtain the first input sequence;

[0018] Input the first input sequence into the video language model, perform context modeling and semantic understanding based on the vector representation, and output segmentation information of the video;

[0019] Split a long video into multiple video clips based on segmentation information.

[0020] As a preferred technical solution, the process of generating the text summary is as follows:

[0021] The video clip is converted into a visual feature representation with semantic information through a visual encoder;

[0022] The visual features are mapped from the original visual space to a space consistent with the input dimension of the large language model through the connection layer to obtain the second input sequence;

[0023] The second input sequence is fed into the video language model, which fuses the visual information with the language information in a unified representation space to obtain a text summary of the video clip.

[0024] As a preferred technical solution, the process of obtaining the key frames of the video clip is as follows:

[0025] Extract the video clip to obtain a frame set;

[0026] Calculate the weight of each frame in the frame set, perform token sampling based on the weight, and obtain a set of video frame tokens;

[0027] The video frame token set and text summary are input into the video language. According to the matching degree between the video frame token and the text summary, the number of frames in the video frame token set that best match the text summary are found.

[0028] As a preferred technical solution, the token sampling consists of two steps: weight calculation and sampling, as follows:

[0029] Use the visual information encoder to obtain the token corresponding to the original video frame , for the token corresponding to the original video frame Compress to get compressed visual token ;

[0030] The attention weights are calculated between the compressed visual tokens of all frames, and the sum of the attention weights of a frame relative to all other frames is the final weight of the frame;

[0031] For frames with high weights, their tokens are converted from the tokens corresponding to the original video frames. For frames with low weights, their tokens are sampled from the compressed visual tokens The corresponding position samples are taken in .

[0032] As a second aspect of the present invention, a sampling system for long video understanding is provided. The system executes the sampling method for long video understanding as described above, including:

[0033] Video segmentation module: takes a long video and video subtitles as input, and uses a large video model to split the long video into multiple semantically consistent video segments;

[0034] Text summary generation module: For each video clip, the visual encoder maps the input sequence to the input sequence, inputs the input sequence into the video language model, and generates a text summary of the video clip;

[0035] Token sampling module: performs token sampling on video clips, calculates the relative weights between frames of the video clips, and assigns different numbers of tokens to each frame according to the weights;

[0036] Key frame acquisition module: The token-sampled video frame and text summary are input into the video language model, and the key frame of the video clip is obtained based on the matching degree between the video frame token and the text summary.

[0037] As a preferred technical solution, the video segmentation module specifically includes:

[0038] Tokenizer, which breaks down the text sequence of the input video subtitles into basic language units;

[0039] Embedding layer, for a text sequence, maps each language unit to a corresponding vector representation;

[0040] The projection layer transforms the vector representation to the input dimension of the video language model to obtain the first input sequence.

[0041] The video language model performs context modeling and semantic understanding based on the vector representation of the first input sequence, and outputs segmented information of the video.

[0042] Video processing tool that divides a long video into multiple video clips based on segmentation information.

[0043] As a preferred technical solution, the text summary generation module specifically includes:

[0044] A visual encoder that converts video clips into visual feature representations with semantic information;

[0045] The connection layer maps the visual features from the original visual space to a space consistent with the input dimension of the large language model to obtain the second input sequence;

[0046] The video language model, for the second input sequence input, fuses visual information and language information in a unified representation space to obtain a text summary of the video clip.

[0047] As a preferred technical solution, the token sampling module specifically includes:

[0048] Token acquisition unit, which uses the visual information encoder to obtain the token corresponding to the original video frame , for the token corresponding to the original video frame Compress to get compressed visual token ;

[0049] The weight calculation unit calculates the attention weights between the compressed visual tokens of all frames, and takes the sum of the attention weights of a frame relative to all other frames as the final weight of the frame;

[0050] Token allocation unit, for frames with high weight, its token is changed from the token corresponding to the original video frame For frames with low weight, the tokens are sampled from the compressed video tokens. The corresponding position samples are taken in .

[0051] As a preferred technical solution, the key frame acquisition module specifically includes:

[0052] A frame extraction unit extracts the video clip to obtain a frame set;

[0053] The sampling unit inputs the video frame token set and text summary obtained by the token sampling module into the video language, and finds the number of frames in the video frame token set that best match the text summary based on the degree of matching between the video frame token and the text summary.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1) The long video comprehension sampling method of the present invention is based on a token sampling method and a large multimodal model. The method of the present invention first segments the long video and ensures the semantic coherence of each segment. Secondly, it generates a text summary of the video segment for each long video segment. Finally, based on the token sampling method, the importance of the frames of all video segments is calculated using the text summary. The number of tokens is dynamically allocated according to the importance of the video frames, thereby improving the efficiency of frame processing while ensuring the integrity of the video information. The method of the present invention can reduce the video memory overhead caused by large-scale video frame processing and realize efficient long video comprehension of a large multimodal model.

[0056] 2) This invention uses a large video language model as its backbone, leveraging its semantic understanding capabilities to segment long videos into multiple semantically consistent segments, avoiding the semantic discontinuity of segments caused by existing solutions that segment videos based on scene changes. Furthermore, the input sequence is fed into the large language model component of the large video language model to generate a textual summary of the video segment. This invention leverages its multimodal capabilities to accomplish understanding tasks across text, image, and video modalities.

[0057] 3) This invention proposes a novel token sampling method that calculates the weight of each frame in a video clip frame set, and then performs token sampling based on the weight. Sampling is performed based on the calculated weight of the compressed video frame, assigning tokens corresponding to the original video frame to frames with high weights, and assigning tokens corresponding to the compressed video frame to frames with low weights. This method addresses the problem of excessive computational resource consumption by large video language models in long video understanding tasks. By using the token sampling method to sample tokens of video frames, this invention can retain complete video information while significantly reducing video memory overhead, thereby improving the performance and efficiency of the model in long video understanding tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of the sampling method for long video understanding proposed by the present invention;

[0059] Figure 2 Schematic diagram of token sampling of the present invention. DETAILED DESCRIPTION

[0060] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0061] Example 1

[0062] The present invention proposes a sampling method for long video understanding, which is based on token sampling and multimodal large models. First, the long video is segmented, and the semantic coherence of each segment is ensured. Secondly, a text summary of the video segment is generated for different long video segments. Finally, based on the token sampling method, the importance of the frames of all video segments is calculated using the text summary, and the number of tokens is dynamically allocated according to the importance of the video frames. While ensuring the integrity of the video information, the frame processing efficiency is improved. The method of the present invention can reduce the video memory overhead caused by large-scale video frame processing and realize efficient long video understanding of multimodal large models. Figure 1 Said method of the present invention comprises the following steps:

[0063] Step S1: Based on the subtitle file of a long video as input, the semantic understanding capability of the multimodal large model is used to divide the long video into multiple semantically consistent video segments.

[0064] In this embodiment, input video subtitles The input sequence is obtained through encoding by the word segmenter and the embedding vector layer. This input sequence is sent to the large language model component in the video large language model to obtain the corresponding output. Specifically, the input text sequence is first processed by the word segmenter and broken down into basic language units (i.e., tokens). The segmented text sequence is then sent to the embedding layer, which maps each token to a corresponding vector representation, i.e., token embedding. These vector representations are then transformed in dimension through a projection layer to adapt to the input dimension requirements of the large language model. The large language model then performs subsequent context modeling and semantic understanding based on these vectors. The present invention utilizes the semantic understanding capability of the large language model to divide a long video into multiple semantically consistent video segments, avoiding the problem of semantic discontinuity of video segments caused by video segmentation based on scene changes in existing technical solutions. The output of the large language model contains the segmentation information of the video large language model, and the long video is divided into multiple video segments based on the segmentation information using video processing tools.

[0065] Step S2: Process each video segment and generate a text summary of the video segment.

[0066] In this embodiment, the input video clip After mapping through the visual encoder and the connection layer, the input sequence is generated. This input sequence is fed into the large language model component of the video large language model to produce the corresponding output. Specifically, the visual encoder processes the video clip, converting it into a visual feature representation with semantic information. The connection layer then maps the visual features from the original visual space to a space consistent with the input dimensions of the large language model, ensuring that visual information can be integrated with language information in a unified representation space. This output contains the video large language model's textual summary of the video clip.

[0067] Step S3: Use the token sampling method to input the sampled video frames into the multimodal large model to improve its long video understanding efficiency.

[0068] In this embodiment, the input is a text summary corresponding to the video clip. and the video clip is uniformly sampled at one frame per second. The token sampling method adopted by the present invention first calculates the frame set Every frame The weight of , and then the token sampling is performed according to this weight, aiming to Identify frame sets The key frame containing the most information is selected to achieve a balance between video memory overhead and video information integrity. Then, the present invention calculates the weight of the compressed video frame. , perform sampling, assign the token corresponding to the original video frame to the frame with high weight, and assign the token corresponding to the compressed video frame to the frame with low weight. Finally, the set of video frame tokens (in or ) and text summary The large language model component that is input to the video large language model enables the large language model to and Make a choice based on the degree of matching and find the set Zhongyu The most appropriate number of frames. Specifically, the large language model selects key frames based on visual and linguistic information. It first uses the relationships between video frames to calculate frames with high information density, retains the number of tokens in these frames, compresses the number of tokens in the remaining frames, and then selects key frames from all processed frames.

[0069] Furthermore, the token sampling method consists of two steps: weight calculation and sampling, aiming to balance the model memory overhead and video information integrity.

[0070] In this embodiment, the input of the token sampling method is a frame set , which is in the form of the token corresponding to the original video frame (in is the number of video frames, is the number of tokens contained in a single original video frame, is the token hidden state size, the same below), the original video frame token is the output of the visual information encoder (mapped by the connection layer). First, for Compress to get compressed visual token (in is the number of tokens contained in a single compressed video frame) In this embodiment, the original video frame token There are 729 tokens per frame, and the 729 tokens per frame are compressed. The specific compression method is to divide the 729 tokens into n parts, and then take the average of the parts, each part gets 1 token, and finally the 729 tokens are compressed into n tokens, that is, compressed visual tokens. The present invention first calculates The relative weight between frames , and then filter out the frames with high relative weights. The process of calculating relative weights is:

[0071]

[0072] in, and for The result after mapping is Respectively represent The compressed visual tokens of all frames calculate attention weights with each other, and the sum of the attention weights of a frame relative to all other frames is the final relative weight of the frame. Figure 2 For convenience, we assume there are five frames. When calculating the weights, we calculate the attention weight of each frame relative to all frames. Then, we average the five attention weights corresponding to each frame to obtain the final attention weight for that frame, which is also known as the relative weight of that frame. In the figure, dark blue represents a high similarity score between two frames, while light blue represents a low similarity score. The average of the scores in the same column represents the relative weight of the frames corresponding to that column. The relative weight of each frame is then calculated based on the calculations for each column, and frames with high relative weights are selected.

[0073] For frames with relatively high weights, their tokens are converted from the tokens corresponding to the original video frames. For frames with low weights, their tokens are sampled from the compressed visual tokens The token sampling method adopted by the present invention can retain the number of tokens of video frames with high information content and reduce the number of tokens of frames with low information content, thereby reducing video memory overhead while retaining the integrity of video information.

[0074] Example 2

[0075] As another embodiment of the present invention, the present invention further provides a sampling system for long video understanding, which executes the sampling method for long video understanding described in Example 1 above, including:

[0076] Video segmentation module: takes a long video and video subtitles as input, and uses a large video model to split the long video into multiple semantically consistent video segments;

[0077] Text summary generation module: For each video clip, the visual encoder maps the input sequence to the input sequence, inputs the input sequence into the video language model, and generates a text summary of the video clip;

[0078] Token sampling module: performs token sampling on video clips, calculates the relative weights between frames of the video clips, and assigns different numbers of tokens to each frame according to the weights;

[0079] Key frame acquisition module: The token-sampled video frame and text summary are input into the video language model, and the key frame of the video clip is obtained based on the matching degree between the video frame token and the text summary.

[0080] Furthermore, the video segmentation module specifically includes: a word segmenter, which splits the text sequence of the input video subtitles into basic language units; an embedding layer, which maps each language unit to a corresponding vector representation for the text sequence; a projection layer, which transforms the dimension of the vector representation and adjusts it to the input dimension of the video large language model to obtain a first input sequence; the video large language model, which performs context modeling and semantic understanding on the first input sequence based on the vector representation and outputs video segmentation information; a video processing tool, which divides a long video into multiple video clips based on the segmentation information.

[0081] Furthermore, the text summary generation module specifically includes: a visual encoder, which converts video clips into visual feature representations with semantic information; a connection layer, which maps visual features from the original visual space to a space consistent with the input dimension of the large language model to obtain a second input sequence; and a video large language model, which, for the second input sequence input, fuses visual information and language information in a unified representation space to obtain a text summary of the video clip.

[0082] Furthermore, the token sampling module specifically includes:

[0083] Token acquisition unit, which uses the visual information encoder to obtain the token corresponding to the original video frame , for the token corresponding to the original video frame Compress to get compressed visual token ;

[0084] The weight calculation unit calculates the attention weights between the compressed visual tokens of all frames, and takes the sum of the attention weights of a frame relative to all other frames as the final weight of the frame;

[0085] Token allocation unit, for frames with high weight, its token is changed from the token corresponding to the original video frame For frames with low weights, their tokens are sampled from the compressed visual tokens The corresponding position samples are taken in .

[0086] Furthermore, the key frame acquisition module specifically includes: a frame extraction unit, which extracts the video clips to obtain a frame set; a sampling unit, which inputs the video frame token set and text summary obtained by the token sampling module into the video language, and finds the number of frames in the video frame token set that best match the text summary based on the degree of matching between the video frame token and the text summary.

[0087] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A sampling method for long video understanding, characterized by the following steps: include: Take a long video and its subtitles as input, and use the video model to split the long video into multiple semantically consistent video segments; For each video clip, the visual encoder is used to map the input sequence, which is then fed into the video language model to generate a text summary of the video clip. Token sampling is performed on the video clip, and the relative weights between the frames of the video clip are calculated. A different number of tokens are assigned to each frame according to the weights. The relative weights are specifically calculated by calculating the attention weights between the compressed visual tokens of all frames, and the sum of the attention weights of a frame relative to all other frames is used as the final weight of the frame. The token-sampled video frames and text summaries are input into the video language model, and the key frames of the video clips are obtained based on the matching degree between the video frame tokens and the text summaries.

2. A sampling method for long video understanding according to claim 1, characterized in that: The process of obtaining the video clip is as follows: The text sequence of the input video subtitles is processed by the word segmenter and split into basic language units; The text sequence passes through the embedding layer, mapping each language unit to a corresponding vector representation; The vector representation undergoes dimension transformation through the projection layer and is adjusted to the input dimension of the video large language model to obtain the first input sequence; Input the first input sequence into the video language model, perform context modeling and semantic understanding based on the vector representation, and output segmentation information of the video; Split a long video into multiple video clips based on segmentation information.

3. The sampling method for long video understanding according to claim 1, characterized in that: The process of generating the text summary is as follows: The video clip is converted into a visual feature representation with semantic information through a visual encoder; The visual features are mapped from the original visual space to a space consistent with the input dimension of the large language model through the connection layer to obtain the second input sequence; The second input sequence is fed into the video language model, which fuses the visual information with the language information in a unified representation space to obtain a text summary of the video clip.

4. The sampling method for long video understanding according to claim 1, characterized in that: The process of obtaining the key frames of the video clip is as follows: Extract the video clip to obtain a frame set; Calculate the weight of each frame in the frame set, perform token sampling based on the weight, and obtain a set of video frame tokens; The video frame token set and text summary are input into the video language. According to the matching degree between the video frame token and the text summary, the number of frames in the video frame token set that best match the text summary are found.

5. The sampling method for long video understanding according to claim 4, characterized in that: The token sampling consists of two steps: weight calculation and sampling, as follows: Use the visual information encoder to obtain the token corresponding to the original video frame , for the token corresponding to the original video frame Compress to get compressed visual token ; The attention weights are calculated between the compressed visual tokens of all frames, and the sum of the attention weights of a frame relative to all other frames is the final weight of the frame; For frames with high weights, their tokens are converted from the tokens corresponding to the original video frames. The corresponding position sampling in ; For frames with low weight, their tokens are converted from compressed visual tokens The corresponding position samples are taken.

6. A sampling system for long video understanding, characterized by: The system executes the sampling method for long video understanding according to any one of claims 1 to 5, including: Video segmentation module: takes a long video and video subtitles as input, and uses a large video model to split the long video into multiple semantically consistent video segments; Text summary generation module: For each video clip, the visual encoder maps the input sequence to the input sequence, inputs the input sequence into the video language model, and generates a text summary of the video clip; Token sampling module: performs token sampling on video clips, calculates the relative weights between frames of the video clips, and assigns different numbers of tokens to each frame according to the weights; Key frame acquisition module: The token-sampled video frames and text summaries are input into the video language model, and the key frames of the video clips are obtained based on the matching degree between the video frame tokens and the text summaries.

7. The sampling system for long video understanding according to claim 6, characterized in that: The video segmentation module specifically includes: Tokenizer, which breaks down the text sequence of the input video subtitles into basic language units; Embedding layer, for a text sequence, maps each language unit to a corresponding vector representation; The projection layer transforms the vector representation to the input dimension of the video language model to obtain the first input sequence. The video language model performs context modeling and semantic understanding based on the vector representation of the first input sequence, and outputs segmented information of the video. Video processing tool that divides a long video into multiple video clips based on segmentation information.

8. The sampling system for long video understanding according to claim 6, characterized in that: The text summary generation module specifically includes: A visual encoder that converts video clips into visual feature representations with semantic information; The connection layer maps the visual features from the original visual space to a space consistent with the input dimension of the large language model to obtain the second input sequence; The video language model, for the second input sequence input, fuses visual information and language information in a unified representation space to obtain a text summary of the video clip.

9. The sampling system for long video understanding according to claim 6, characterized in that: The token sampling module specifically includes: Token acquisition unit, which uses the visual information encoder to obtain the token corresponding to the original video frame , for the token corresponding to the original video frame Compress to get compressed visual token ; The weight calculation unit calculates the attention weights between the compressed visual tokens of all frames, and takes the sum of the attention weights of a frame relative to all other frames as the final weight of the frame; Token allocation unit, for frames with high weight, its token is changed from the token corresponding to the original video frame For frames with low weight, the tokens are sampled from the compressed video tokens. The corresponding position samples are taken.

10. The sampling system for long video understanding according to claim 9, characterized in that: The key frame acquisition module specifically includes: A frame extraction unit extracts the video clip to obtain a frame set; The sampling unit inputs the video frame token set and text summary obtained by the token sampling module into the video language, and finds the number of frames in the video frame token set that best match the text summary based on the degree of matching between the video frame token and the text summary.

Images