A video level structure-based adaptive key frame sampling method and system for long video understanding

Abstract

The application discloses a long video understanding-oriented adaptive key frame sampling method and system based on a video hierarchical structure, which comprises the following steps: performing hierarchical structure splitting on a long video to be processed; calling a video large language model component to generate summaries of semantic event segments, scene segments and shot segments; calling the large language model component according to a task text to calculate similarity scores of each level and comprehensive similarity scores of each shot segment; screening top K shot segments as candidate sampling units; under the constraint of a preset total sampling frame number, calculating sampling weights and frame quotas of each candidate sampling unit; and performing adaptive key frame sampling in the corresponding candidate sampling unit. The application solves the problems of poor performance of existing long video understanding methods in key frame selection tasks and consumption of a large amount of computing resources, and is suitable for video content labeling, summary generation and question and answer tasks driven by a video large language model.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a keyframe sampling method and system for long video structured understanding tasks. Background Technology

[0002] Long-video understanding tasks based on large models refer to tasks where, given a lengthy video as input, the large model can provide structured annotations and summaries of the video content in text form, and offer video-related answers based on user questions. The main challenge of large models for long-video understanding is the limited context length, which prevents them from processing all video frames. Therefore, accurately selecting appropriate video segments or keyframes is crucial. Keyframe selection is a core step in generating high-quality video understanding, directly determining the conciseness and information carrying capacity of the video content. Therefore, within a limited context window, intelligently selecting the most informative keyframes maximizes the video language model's ability to understand long videos. Existing video keyframe sampling techniques mainly include uniform frame extraction, motion analysis-based adaptive sampling, query-aware sampling, and differential sampling. These methods generally suffer from a trade-off between information redundancy and loss of key information when processing long videos: uniform sampling cannot adapt to dynamic changes in video content; motion analysis-based methods rely excessively on physical motion features, ignoring static but important semantic information; query-aware methods, while improving relevance, are highly dependent on query quality, computationally expensive, and prone to oversampling of locally relevant segments; differential sampling, while reducing redundancy, has complex redundancy calculations and poor adaptability to progressively changing scenes. Therefore, a novel keyframe sampling mechanism that balances information integrity and computational efficiency, possesses task-aware capabilities, and is computationally efficient is urgently needed. Summary of the Invention

[0003] The purpose of this invention is to overcome the problems of poor performance and high computational resource consumption in existing long video understanding methods, and to provide an adaptive keyframe sampling method based on video hierarchy structure for long video understanding, which is applicable to video content annotation, summarization and question answering tasks driven by video large language model.

[0004] The objective of this invention can be achieved through the following technical solutions: In a first aspect, the present invention provides an adaptive keyframe sampling method based on video hierarchy structure for long video understanding, comprising the following steps: Step 1: Decompose the long video to be processed into a hierarchical structure and construct a video hierarchy structure set containing semantic event segments, scene segments, and shot segments. Step 2: The semantic event segments, scene segments, and shot segments in the video hierarchical structure set are mapped to the input sequence by the visual encoder, and the video big language model component is called to generate text summaries of the corresponding levels, so as to obtain semantic event segment summaries, scene segment summaries, and shot segment summaries, and store them in the structured summary library. Step 3: Receive the task text for long video understanding, call the large language model component, and calculate the semantic similarity between the task text and the semantic event segment summary, scene segment summary and shot segment summary in the structured summary library respectively to obtain the similarity scores at three levels; Step 4: Based on the similarity scores of the three levels, calculate the comprehensive similarity score of each shot segment, sort all shot segments according to the comprehensive similarity score, and select the top K shot segments as candidate sampling units. Step 5: Under the constraint of the preset total number of sampling frames X, calculate the sampling weight and frame quota of each candidate sampling unit according to the comprehensive similarity score and shot length, and perform adaptive keyframe sampling in the corresponding candidate sampling unit according to the frame quota.

[0005] In a second aspect, the present invention also provides an adaptive keyframe sampling system based on video hierarchy for long video understanding, the system being configured to perform the method described in the first aspect, the system comprising: The video hierarchical structured parsing module is used to receive long video data to be processed and to split the long video into a hierarchical set containing semantic event segments, scene segments and shot segments through the semantic event detection unit, scene detection unit and shot detection unit. The segment summarization and summarization library construction module is used to map video segments of each level into visual feature sequences through a visual encoding unit, generate summary texts of each level through a video big language model interaction interface unit, and store the summary texts and timestamp indexes through a structured summary library unit. The task-aware semantic matching module is used to receive the task text input by the user, call the large language model component to calculate the semantic similarity between the task text and the summaries at each level in the structured summary library, and generate similarity scores at three levels. The comprehensive screening and adaptive sampling control module is used to calculate the comprehensive similarity score of the shot segments based on the similarity scores of the three levels, screen the top K candidate shots, and calculate the sampling frame quota for each candidate shot under the constraint of the total number of sampling frames, combining the shot length and the comprehensive similarity score, and perform sampling to output the keyframe sequence.

[0006] In one embodiment, the video large language model component and the large language model component are configured as a pre-trained general model interface; The interface supports remote invocation via cloud application programming interface (API) or loading and running via local computing unit; The system is configured to use the interface as an independent inference tool to receive natural language descriptions or semantic similarity scores output by the component to execute sampling logic, and the execution of the sampling logic does not depend on the specific model network architecture inside the component.

[0007] In one embodiment, the semantic event detection unit, scene detection unit, and camera detection unit are configured as replaceable visual analysis algorithm modules; The module is configured to integrate any existing computer vision algorithm or pre-trained model with corresponding detection capabilities, receive video data streams through a standardized data interface and output time boundary information including start and end timestamps; The system constructs the hierarchical set based solely on the temporal boundary information output by the module, without limiting the specific feature extraction method or network model architecture used within the module.

[0008] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0009] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect. Alternatively, the present invention also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the method of the above-described method embodiments.

[0010] The beneficial effects of this invention include: 1. Existing technologies often force the truncation or excessive sparse sampling of long videos due to contextual constraints, resulting in the loss of key information. This invention uses a three-level structure of "semantics-scene-shot" to guide the sampling of lower levels with upper-level semantics and combines a Top-K filtering mechanism to remove irrelevant noise, ensuring that the core information most relevant to the user's task is preserved to the maximum extent within a limited total number of sampling frames (X).

[0011] 2. This invention achieves a highly efficient processing mode of "one-time computation, multiple reuses," significantly reducing computational resource consumption. It creatively decouples the video processing workflow into two stages: offline preprocessing and online task sampling. For the same long video segment, complex video splitting and summary generation (steps 1-S02) only need to be performed once and stored in the summary library; when users raise different questions, only low-overhead text similarity calculation and sampling allocation are required (steps 3-S05). Compared to traditional methods that require rescanning the video content for each task, this system greatly saves GPU computing power and time costs.

[0012] 3. An adaptive sampling algorithm based on comprehensive weights and length is proposed, solving the problem of uneven sampling between long and short shots. Existing query-aware sampling methods often ignore the duration attribute of the shot itself, easily leading to oversampling of short shots or undersampling of long shots. The dynamic quota allocation formula proposed in this invention (combining weights) With length By introducing a sensitivity coefficient α and a safety net mechanism It intelligently strikes a balance between "high relevance" and "information richness," ensuring that important long shots receive more frames while short shots also get the necessary exposure, thereby improving the accuracy of downstream video understanding tasks. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the long video hierarchical structure of the present invention.

[0015] Figure 2 This is a schematic diagram of the algorithm steps of an adaptive keyframe sampling method based on video hierarchy structure for long video understanding according to the present invention.

[0016] Figure 3 This is a schematic diagram of an adaptive keyframe sampling system architecture based on video hierarchy structure for long video understanding, according to the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0018] This invention provides an adaptive keyframe sampling method based on video hierarchy for long video understanding. It generates hierarchical video summaries through preprocessing and dynamically selects keyframes with task awareness, solving the problems of key information loss and computational resource waste caused by the limitation of large model context length in existing technologies.

[0019] In this invention, the video hierarchy structure is as follows: Figure 1 As shown, long videos typically contain multiple sub-events, composed of multiple semantic events / scenes / shots. A long video can be divided from top to bottom into: semantic event segments, scene segments, shot segments, and keyframes. Semantic event segments: video segments with independent narrative functions (e.g., a 'proposal'). Scene segments: video segments composed of a single, continuous spatiotemporal background (e.g., a 'kitchen conversation'). Shot segments: a sequence of consecutive frames captured in a single shot by a camera (without editing points). Keyframes: a number of image frames that best represent the information of a video segment.

[0020] refer to Figure 2 The sampling method of this invention for long video understanding mainly includes the following steps: S01: Taking the long video to be processed as input, the long video is decomposed into a hierarchical structure using a hierarchical detection algorithm, constructing a video hierarchical structure set containing semantic event segments, scene segments, and shot segments. Clearly, the long video contains multiple semantic event segments, each semantic event segment contains at least one scene segment, and each scene segment contains at least one shot segment. For example, the hierarchical structure decomposition in this step may specifically include: obtaining semantic event segments using a general event boundary detection algorithm, obtaining scene segments using a scene segmentation algorithm, and obtaining shot segments using a shot detection algorithm.

[0021] S02: The semantic event fragments, scene fragments, and shot fragments are mapped into input sequences by a visual encoder, and the video big language model component is called to generate text summaries of the corresponding levels, namely semantic event fragment summaries, scene fragment summaries, and shot fragment summaries; and the resulting summaries are stored in a structured summary library.

[0022] S03: Receive the task text input by the user for long video understanding, call the large language model component, and calculate the semantic similarity between the task text and the semantic event segment summaries, scene segment summaries, and shot segment summaries in the structured summarization library, respectively, to obtain three levels of similarity scores, namely: semantic event similarity. Scene similarity Similarity to the lens For example, this step calls the large language model component to calculate semantic similarity, specifically by calculating the cosine similarity between the task text and the feature vectors of the summary text at each level.

[0023] S04: Based on the similarity scores of the three levels, calculate the comprehensive similarity score of each shot segment using a comprehensive similarity calculation method. ; and according to All shot clips are sorted, and the top K shot clips are selected as candidate sampling units.

[0024] For example, in this step, a weighted fusion method or a hierarchical gating adjustment method can be used to calculate the overall similarity score for each shot segment.

[0025] The calculation formula for the weighted fusion method is as follows: In the formula, The overall similarity score for the shot segments is calculated. , , For weight parameters, and ; In the formula, the calculation formula for the hierarchical gating adjustment method is: in, , This is the adjustment coefficient.

[0026] S05: Under the constraint of the preset total number of sampling frames X, based on the comprehensive similarity score... Given the shot length L, calculate the sampling weight and frame quota for each candidate sampling unit, and perform adaptive keyframe sampling within the corresponding candidate sampling unit according to the frame quota.

[0027] Furthermore, in step 5, the adaptive keyframe sampling specifically includes: First, set the total number of sampling frames X allowed by the video large language model.

[0028] Secondly, calculate each candidate shot Allocation weights The calculation formula is: in, The frame length of this shot. The overall similarity score for the shot is given by α, which is a preset sensitivity coefficient (preferably α≥1 to strengthen the weight of highly relevant shots).

[0029] Next, calculate each candidate shot. sampling allocation The calculation formula is: in, Minimum guaranteed sampling number for each candidate shot, preferred ≥1, further preferred , This indicates rounding down to the nearest integer.

[0030] Finally, based on the calculation In each candidate shot Uniform sampling is performed within the time range to obtain an X-frame keyframe sequence that maximizes the coverage of key information in the long video and meets the total computational constraints.

[0031] S01-S02 can be summarized as long video preprocessing, requiring only one processing session for each long video. S03-S05 can be summarized as adaptive keyframe sampling, selectively acquiring keyframes for each different long video understanding task text. This achieves a balance between computational accuracy and resource consumption. The text summary for all video segments only needs to be calculated once, and the summaries for all video segments are saved. For each new task, based on the specific long video understanding task text, the optimal targeted video segments can be quickly selected, and the keyframe sampling density can be adaptively controlled.

[0032] A second aspect of the present invention provides an adaptive keyframe sampling system based on video hierarchy for long video understanding, the system being configured to execute the aforementioned adaptive keyframe sampling method based on video hierarchy for long video understanding, as follows: Figure 3 As shown, the system includes: The video hierarchical structured parsing module, used to execute S01, receives long video data to be processed, including: The semantic event detection unit uses a semantic event detection algorithm to break down a long video into a set of semantic event segments; The scene detection unit uses scene detection algorithms to break down long videos into sets of scene segments; The shot detection unit uses shot detection algorithms to break down long videos into sets of shot segments; The fragment summarization and summarization library construction module is used to execute S02. This module includes: Visual coding unit, used to map video segments at each level into a sequence of visual features; The video big language model interaction interface unit is configured to call a pre-trained video big language model component. The component runs through an application programming interface (API) or local deployment and is used to receive visual feature sequences and generate corresponding semantic event fragment summaries, scene fragment summaries, and shot fragment summaries. The structured summary library unit is used to store the generated summary texts at each level and their corresponding timestamp indexes for reuse in subsequent tasks.

[0033] The task-aware semantic matching module is used to execute S03. This module is configured to receive the task text input by the user and call the large language model component to calculate the semantic similarity between the task text and the summaries at each level stored in the structured knowledge base, and generate semantic similarity, scene similarity and shot similarity scores.

[0034] The large language model component runs via an application programming interface (API) or local deployment, and is used to receive the task text and text information stored in the structured summary library and generate corresponding semantic similarity.

[0035] The integrated screening and adaptive sampling control module, used to execute S04 and S05, is the core computing unit and specifically includes: The comprehensive scoring unit is used to calculate the comprehensive similarity score of each shot segment based on a preset fusion algorithm (such as weighted fusion or gating adjustment). ; The candidate shot selection unit is used to sort all shot segments and select the Top-K highly relevant shots as candidate sampling units; The sampling frame quota allocation unit is used to allocate quotas according to the formula, under the constraint of the total number of sampling frames X. Calculate the assigned weights and apply them according to the formula. Calculate the sampling frame quota for each shot; The sampling execution unit is used to allocate quotas based on the calculated frame count. Uniform sampling is performed within the time range of the corresponding shot to output the final keyframe sequence.

[0036] The following are some specific embodiments of the present invention.

[0037] Example 1 An adaptive keyframe sampling method based on video hierarchy for long video understanding, such as... Figure 1 and Figure 2 As shown, this embodiment provides a sampling method for long video understanding. This method decouples the video processing flow into an "offline preprocessing stage" and an "online task sampling stage".

[0038] The offline preprocessing stage (corresponding to steps 1-2) aims to transform unstructured long videos into a structured semantic summary library.

[0039] Step 1: Take the long video to be processed as input, and use a hierarchical detection algorithm to split the long video into hierarchical structures, constructing a hierarchical set containing semantic event segments, scene segments, and shot segments; wherein, each semantic event segment contains at least one scene segment, and each scene segment contains at least one shot segment; In this embodiment, a 2-hour movie video file is input, and the system uses a bottom-up strategy to construct the video hierarchy structure: Shot detection: The TransNet V2 deep learning model is used as the shot detection algorithm to identify hard cuts and gradual transitions in the video, and to segment the video into a set of M shot segments. .

[0040] Scene detection: The LGSS (Local-Global Scene Segmentation) algorithm is used. Based on the visual feature similarity and temporal proximity of keyframes, multiple shots with consistent visual styles and continuous spatiotemporal backgrounds are clustered into a scene segment. The final scene set is obtained. .

[0041] Semantic event detection: Using the GEBD (Generic Event Boundary Detection) algorithm, the boundaries of semantic event segments with consistent narrative themes in the complete video are detected, ultimately yielding an event set. .

[0042] This constructs a three-level tree structure of "semantic event-scene-shot".

[0043] Step 2: Map the semantic event fragments, scene fragments, and shot fragments into input sequences using a visual encoder, and call the video big language model component to generate text summaries at the corresponding levels, namely semantic event fragment summaries, scene fragment summaries, and shot fragment summaries; and store the summaries in a structured summary library.

[0044] The system calls a pre-trained video large language model component (in this embodiment, open-source models such as Video-LLaMA or LLaVA-Video can be selected).

[0045] Visual encoding: Visual feature vectors for each segment are extracted using a pre-trained CLIP-ViT-L / 14 visual encoder.

[0046] Summary generation: Visual feature sequences are input into the video large language model to generate text summaries at different granularities. For example, for shot clips, specific action descriptions are generated (e.g., "A man is running in the rain"); for scene clips, descriptions of environment and atmosphere are generated (e.g., "Rainy night street, tense atmosphere"); for semantic event clips, plot descriptions are generated (e.g., "The male protagonist tries to escape pursuit").

[0047] Storage: Store the generated summary text, the corresponding timestamps (Start_time, End_time), and the hierarchical relationship into a structured summary library.

[0048] The online task sampling stage (corresponding to steps 3-5) aims to select the most matching video frame sequence in real time based on the user's specific questions.

[0049] Step 3: Receive the task text input by the user for long video understanding, call the large language model component, calculate the semantic similarity between the task text and the semantic event segment summary, scene segment summary and shot segment summary in the structured summary library, and obtain the similarity scores at three levels. Assuming the user inputs the task text (Query) as: "Find the scene where the protagonist falls in the rain", the system calls the text large model component (or uses the Deepseek interface) to calculate the cosine similarity between the Query and all summaries in the summary database, obtaining three levels of similarity scores: semantic event similarity ( Scene similarity ( ), lens similarity ( ).

[0050] Step 4: Based on the similarity scores of the three levels, calculate the comprehensive similarity score for each shot segment using a comprehensive similarity calculation method. ; and according to All shot clips are sorted, and the top K shot clips are selected as candidate sampling units; In this embodiment, the comprehensive similarity calculation method can be either a weighted fusion method or a hierarchical gating adjustment method; The weighted fusion method calculates the weights by combining "local details (shots)," "mid-level background (scenes)," and "global narrative (semantic events)." The specific calculation formula is as follows: Among them, setting weight parameters , , ,satisfy In this embodiment, the parameter settings are used to highlight the importance of the details of the lens itself, while using scene and event information to smooth the similarity.

[0051] Hierarchical gating control method, with As the underlying similarity score, if the upper-level (semantic events / scenes) is completely unrelated to the task, then even if the lower-level shots have high similarity, they may still be misclassified. The upper-level score is used as a coefficient to scale the lower-level score; the specific calculation formula is as follows: Among them, setting weight parameters , In this embodiment, the parameter settings are used to limit the similarity weights of scenarios and semantic events.

[0052] After the calculations are completed, apply the following to all shots: Sorting in descending order, the top K=10 shots are selected as candidate sampling units.

[0053] Step 5: Under the constraint of the preset total number of sampling frames X, calculate the results based on the comprehensive similarity score. Given the shot length L, calculate the sampling weight and frame quota for each candidate sampling unit, and perform adaptive keyframe sampling.

[0054] The system is preset to allow a total of X = 64 frames for sampling the downstream large model. To maximize information within this limited number of frames, adaptive keyframe sampling is performed. Calculate each candidate shot Allocation weights The calculation formula is: Setting α=2 ensures that highly relevant shots receive more attention; Calculate each candidate shot sampling allocation The calculation formula is: set up =1 ensures that the minimum guaranteed sampling frame count for each candidate shot is 1 frame; According to the calculation In each candidate shot Uniform sampling is performed within the time range, and the X frames obtained from these K shots are stitched together in chronological order. This stitched image is then used as the final long video keyframe sampling result and input into the downstream large model for question and answer generation.

[0055] Example 2 An adaptive keyframe sampling system based on video hierarchy for long video understanding, such as... Figure 3 As shown, this embodiment provides a system architecture for executing the above method. The system is server-based and includes the following core modules: Video Hierarchical Structured Parsing Module: This module deploys TransNet V2, LGSS, and GEBD algorithm scripts. When long video data (such as MP4 format) is received, it automatically executes the video hierarchical structure splitting logic. This module supports GPU acceleration (such as NVIDIA RTX 4090).

[0056] Fragment summarization and summarization library building module: Visual coding unit: Runs CLIP visual encoder to convert video frames into 512-dimensional or 768-dimensional feature vectors.

[0057] The Video Large Language Model Interaction Interface Unit encapsulates the calling logic for Video-LLM. In its implementation, if local GPU memory is sufficient, a quantized version of the Video-LLaMA-7B-int4 model can be deployed; if local resources are limited, an HTTPS request is made to call the cloud-based GPT-4o or Gemini API interface to transmit the visual token and receive the text description.

[0058] Structured summary library unit: MySQL is used to store metadata (ID, Start_time, End_time, Text), and Milvus vector database is used to store text summary embedding vectors for fast retrieval.

[0059] Task-Aware Semantic Matching Module: This module provides a RESTful API interface to receive query text from front-end users. Internally, it integrates a large language model API interface, used to call the large language model to calculate the semantic similarity between the text and summaries in the library.

[0060] Comprehensive Filtering and Adaptive Sampling Control Module: This module is the core calculation unit for comprehensive shot similarity and keyframe sampling of long videos, and can be implemented using calculation logic written in Python; The system's logical control center calculates similarity and is implemented using computational logic written in Python.

[0061] The comprehensive scoring calculation unit is responsible for evaluating the lens. The calculation is performed; the candidate shot selection unit is responsible for quickly selecting the K shots with the highest scores; the sampling frame quota allocation unit implements the mathematical formula described in Embodiment 1. and The execution unit calls the FFmpeg tool library to quickly extract keyframes from the original video based on the calculated precise timestamps and frame counts, and combines them into a keyframe set.

[0062] It should be noted that the "large language model" or "video large language model" in this specification is a data processing component that plays a role in feature extraction and semantic conversion in this system. This invention does not limit the specific model version (such as GPT series, LLaMA series, Qwen series, etc.), and any pre-trained model with image and text understanding capabilities and text generation capabilities can be applied to this system.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; without departing from the technical solutions of the present invention, all such modifications or substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A video hierarchy based adaptive key frame sampling method for long video understanding, characterized in that, Includes the following steps: Step 1: Decompose the long video to be processed into a hierarchical structure and construct a video hierarchy structure set containing semantic event segments, scene segments, and shot segments. Step 2: The semantic event segments, scene segments, and shot segments in the video hierarchical structure set are mapped to the input sequence by the visual encoder, and the video big language model component is called to generate text summaries of the corresponding levels, so as to obtain semantic event segment summaries, scene segment summaries, and shot segment summaries, and store them in the structured summary library. Step 3: Receive the task text for long video understanding, call the large language model component, and calculate the semantic similarity between the task text and the semantic event segment summary, scene segment summary and shot segment summary in the structured summary library respectively to obtain the similarity scores at three levels; Step 4: Based on the similarity scores of the three levels, calculate the comprehensive similarity score of each shot segment, sort all shot segments according to the comprehensive similarity score, and select the top K shot segments as candidate sampling units. Step 5: Under the constraint of the preset total number of sampling frames X, calculate the sampling weight and frame quota of each candidate sampling unit according to the comprehensive similarity score and shot length, and perform adaptive keyframe sampling in the corresponding candidate sampling unit according to the frame quota.

2. The method of claim 1, wherein the video hierarchy-based adaptive key frame sampling method for long video understanding is characterized by, In step 1, the long video to be processed is split into layers by a hierarchical detection algorithm, including: obtaining semantic event segments using a general event boundary detection algorithm, obtaining scene segments using a scene segmentation algorithm, and obtaining shot segments using a shot detection algorithm. The construction relationship of the video hierarchy set is as follows: the long video contains multiple semantic event segments, each semantic event segment contains at least one scene segment, and each scene segment contains at least one shot segment.

3. The adaptive keyframe sampling method based on video hierarchy structure for long video understanding according to claim 1, characterized in that, In step 3, the similarity scores at the three levels include: semantic event similarity. Scene similarity Similarity to the lens ; The step of calling the large language model component to calculate semantic similarity specifically involves calculating the cosine similarity between the feature vectors of the task text and the summary text at each level.

4. The adaptive keyframe sampling method based on video hierarchy structure for long video understanding according to claim 3, characterized in that, In step 4, a weighted fusion method or a hierarchical gating adjustment method is used to calculate the overall similarity score of each shot segment; The calculation formula for the weighted fusion method is as follows: in, The overall similarity score for the shot segments is calculated. , , For weight parameters, and ; The calculation formula for the hierarchical gating adjustment method is as follows: in, , This is the adjustment coefficient.

5. The adaptive keyframe sampling method based on video hierarchy structure for long video understanding according to claim 1, characterized in that, In step 5, the adaptive keyframe sampling specifically includes the following sub-steps: Set the total number of sampling frames allowed by the large language model of the video, X; Calculate each candidate shot Allocation weights The calculation formula is: in, The frame length of this shot. The overall similarity score for this shot is given by α, where α is the preset sensitivity coefficient. Calculate each candidate shot sampling allocation The calculation formula is: in, The minimum guaranteed number of samples for each candidate shot. Indicates rounding down to the nearest integer; According to the calculation In each candidate shot Uniform sampling is performed within a time range to obtain a keyframe sequence.

6. The adaptive keyframe sampling method based on video hierarchy structure for long video understanding according to claim 5, characterized in that, The sensitivity coefficient α ≥ 1, the minimum guaranteed sampling number ≥1.

7. An adaptive keyframe sampling system based on video hierarchy for long video understanding, characterized in that, The system is configured to perform the method according to any one of claims 1 to 6, the system comprising: The video hierarchical structured parsing module is used to receive long video data to be processed and to split the long video into a hierarchical set containing semantic event segments, scene segments and shot segments through the semantic event detection unit, scene detection unit and shot detection unit. The segment summarization and summarization library construction module is used to map video segments of each level into visual feature sequences through a visual encoding unit, generate summary texts of each level through a video big language model interaction interface unit, and store the summary texts and timestamp indexes through a structured summary library unit. The task-aware semantic matching module is used to receive the task text input by the user, call the large language model component to calculate the semantic similarity between the task text and the summaries at each level in the structured summary library, and generate similarity scores at three levels. The comprehensive screening and adaptive sampling control module is used to calculate the comprehensive similarity score of the shot segments based on the similarity scores of the three levels, screen the top K candidate shots, and calculate the sampling frame quota for each candidate shot under the constraint of the total number of sampling frames, combining the shot length and the comprehensive similarity score, and perform sampling to output the keyframe sequence.

8. The adaptive keyframe sampling system based on video hierarchy for long video understanding according to claim 7, characterized in that, The video large language model component and the large language model component are configured as pre-trained general model interfaces; The interface supports remote invocation via cloud application programming interface (API) or loading and running via local computing unit; The system is configured to use the interface as an independent inference tool to receive natural language descriptions or semantic similarity scores output by the component to execute sampling logic, and the execution of the sampling logic does not depend on the specific model network architecture inside the component.

9. The adaptive keyframe sampling system based on video hierarchy for long video understanding according to claim 7, characterized in that, The semantic event detection unit, scene detection unit, and camera detection unit are configured as replaceable visual analysis algorithm modules; The module is configured to integrate any existing computer vision algorithm or pre-trained model with corresponding detection capabilities, receive video data streams through a standardized data interface and output time boundary information including start and end timestamps; The system constructs the hierarchical set based solely on the temporal boundary information output by the module, without limiting the specific feature extraction method or network model architecture used within the module.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

Images