A large model assisted video summarization method based on global entity unification and robust importance scoring

Abstract

The application discloses a large model assisted video summary method based on global entity unification and robust importance scoring. The method aims at the problems of entity identity fragmentation and unstable matching existing in the "video-text-video" video summary paradigm, and constructs a large model assisted framework integrating global entity unification and robust importance scoring: first, frame-level descriptions containing entity labels are generated, then cross-frame entity unification is completed and global IDs are assigned, and a coherent entity-aware description text set is obtained; after visual-text cross verification and cleaning of the description text set, a double-layer text summary of shot level and video level is generated, and then a multi-granularity mechanism of fine-grained frame-shot scoring and coarse-grained shot-video scoring is used to calculate the robust importance score, key shots are screened and spliced to obtain a video summary. The application can effectively improve the accuracy and robustness of the video summary, improve the video semantic understanding ability and key frame recognition ability, and promote the application in complex scenes.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and natural language processing, and relates to video summarization technology. Specifically, it relates to a large model-assisted video summarization method based on global entity unification and robust importance scoring, which is applicable to scenarios such as video content analysis, intelligent video retrieval, short video generation, and intelligent monitoring. Background Technology

[0002] With the explosive growth of video data, video summarization technology has become crucial for the efficient browsing, retrieval, and transmission of video content. Its core objective is to condense a complete video into a concise representation containing core information, extracting key segments and salient information from the video. Existing video summarization methods have several limitations: first, they heavily rely on manual annotation, a time-consuming, labor-intensive, and subjective process; second, they have poor generalization ability, with models trained in specific domains performing poorly in other domains; and third, relying solely on low-level visual features, they cannot achieve high-level semantic depth understanding of complex video narratives.

[0003] Currently, deep learning-based video summarization methods can be mainly classified into the following categories: (1) There are supervised methods Supervised methods treat video summarizing as a structured prediction problem, attempting to estimate the importance of each frame by modeling the temporal dependencies between frames. During training, the model takes a sequence of frames and labeled data indicating the importance of each frame as input, models the temporal dependencies between frames, and outputs the importance of each frame. Then, the model's predicted importance is compared with the ground truth labels, a loss function is calculated, and the model's training is supervised. While these methods can achieve high summarizing accuracy, creating frame-level ground truth labels requires significant manpower and time, limiting the size of the dataset and inevitably introducing a degree of subjectivity.

[0004] (2) Weak supervision method Weakly supervised methods attempt to reduce the need for manually labeled frame-level importance tags. These methods do not use fine-grained frame-level tags, but instead use weaker tags that are less costly to obtain, such as video-level metadata, importance tags for a small subset of frames, and segment-level importance tags. However, these methods are limited by the fuzziness of the labeled data, resulting in limited generalization ability of the model.

[0005] (3) Unsupervised methods To address the aforementioned issues, researchers have attempted to train models directly on the original video dataset without using any ground truth labels. These methods are primarily categorized into generative adversarial learning (GAP)-based and reinforcement learning-based methods. The former typically consists of a keyframe selector, a generator, and a discriminator. The keyframe selector estimates the importance of frames and generates summary videos, while the generator reconstructs the original videos from the summary videos. The discriminator distinguishes between the reconstructed original videos and the ground truth videos. The latter uses reinforcement learning to train the model to generate summary videos with optimal specific attributes. An evaluator quantifies the presence of specific desired features in the summary videos based on a reward function, which guides the model's training. However, these methods rely solely on video pixel features and cannot capture the semantics of the video.

[0006] (4) Multimodal method Multimodal methods have been investigated to explore the feasibility of combining data from modalities other than vision to complete video summarization tasks. These other modalities include audio and text data. Taking a video summarization method that incorporates text data (such as video titles and descriptions) as an example, during training, the model takes video frame sequences, text data, and ground truth labels (supervised and weakly supervised methods) as input to estimate the importance of frames and generate a summarized video.

[0007] (5) End-to-end method based on large model In recent years, end-to-end multimodal large models have been introduced into video summarization tasks. These methods either directly utilize large models to evaluate frame importance or first generate descriptive text for the video, and then the large model generates a summary text that summarizes the video content. Finally, the semantic distance between the descriptive text and the summary text for each frame is calculated, and the frame corresponding to the descriptive text that is semantically similar to the summary text is selected as the keyframe. However, the above methods have two technical drawbacks: First, the fragmentation problem of entity identity: frame-level descriptions are generated independently, and the same person, object, or other entity in different frames is often referred to by vague pronouns. The model cannot establish cross-frame entity associations, making it difficult to understand the long-term narrative and entity relationships of the video, resulting in logical breaks and semantic incoherence in the summary; Second, the instability problem of matching: when relying on the semantic similarity between the text summary and the frame description to select keyframes, even slight changes in the text wording can cause drastic fluctuations in importance scores, making keyframe selection unstable and the summary results lack representativeness.

[0008] Overall, existing video methods have limitations to varying degrees: supervised methods are limited by the size of the dataset; weakly supervised methods are limited by insufficient annotation information and cannot guarantee accuracy; unsupervised methods cannot extract high-level semantic information from videos; and methods based on large models are hampered by fragmented entity identities and unstable matching, making it difficult to generate coherent, stable, and accurate video summaries. Summary of the Invention

[0009] To address the issues of entity identity fragmentation and matching instability in existing "video-text-video" paradigm video summarization methods, this invention provides a large-model-assisted video summarization method based on global entity unification and robust importance scoring. This method solves the entity identity fragmentation problem by constructing a global entity unification (GEU) mechanism and the matching instability problem by constructing a robust importance scoring (RIS) mechanism. This approach unifies entity identities, improves matching stability, enhances the coherence and representativeness of video summarization, and maintains strong semantic understanding capabilities, thereby improving the accuracy and stability of video summarization generation.

[0010] To achieve the above-mentioned objectives, this invention provides a large-model-assisted video summarization method based on global entity unification and robust importance scoring, comprising the following steps: Video frame sampling and entity-aware descriptive text generation: ① Perform frame sampling on the input raw video to obtain a discrete video frame sequence; ② A visual language model is used to generate entity-aware video descriptions for each sampled frame, generating frame-level local description text that includes key entities and their identifiable attributes.

[0011] Furthermore, global entity unification processing of the description text is completed. MapReduce logic is used to perform global entity unification processing on frame-level local description texts, resulting in a unified entity-aware description text set, specifically including: ① Map phase: Generate frame-level local description text containing entity local labels for each sampled frame; ② Reduce stage: Aggregate fragmented entity information in the video time sequence, identify the same entity that appears repeatedly in different frames using a visual language model, and assign a standardized entity to each entity. <entity> <index>A unified global ID format is used to update the original frame-level local description text, eliminating entity name duplication and semantic loss issues, and forming a unified entity-aware description text set.

[0012] Furthermore, the descriptive text is cleaned: the frame-level descriptive text after global entity uniform processing is cleaned to filter out noise and irrelevant information, resulting in a cleaned descriptive text set that is semantically consistent with the video visual content. The specific steps are as follows: ① The image encoder using the CLIP model encodes all sampled frames to obtain frame-level visual features; ② Based on frame-level visual features, the KTS algorithm is used to segment the original video into several non-overlapping video shots that are semantically close internally and sparse externally; ③ For any sampled frame in each video shot, the cosine similarity between the frame-level visual features and the text features of all descriptive texts in the shot is calculated, and the descriptive text with the highest similarity is selected as the cleaned descriptive text for that frame.

[0013] Furthermore, summary texts for shots and videos are generated. A large model is used to generate two-level summary texts at the shot and video levels from the cleaned descriptive text set, resulting in shot-level summary sets and video-level global summaries, specifically including: ① Shot-level text summarization: The descriptive text of all sampled frames of each shot is concatenated into the descriptive text of the shot, and a large language model is used to generate the summary text of each shot; ② Video-level global text summarization: All shot-level summaries are concatenated into a video description text, and a large language model is used to generate video-level summary text.

[0014] Furthermore, importance scores are calculated for each frame, combining visual features, shot-level summary text, and video-level global summary text to perform multi-granularity importance score calculations, including: ① Frame-Shot Scoring: Calculate the semantic similarity between a sampled frame and the shot-level summary text of its corresponding shot; ② Shot-Video Rating: Calculate the semantic similarity between the shot-level summary text and the video-level summary text for each video shot; Furthermore, the frame sampling rate is 3fps; the visual language model generates the prompt "Please describe this image in one short sentence, like person a ...", which makes the generated description text contain entity local labels to annotate each entity appearing in the video.

[0015] Furthermore, the visual language model used is Qwen3.5-Plus, and the large model used is Tencent Hunyuan Large Model.

[0016] Furthermore, to complete the final selection of key shots and the generation of video summaries, a 0 / 1 knapsack algorithm is used. Based on the frame importance score, key video shots are selected from the original video, and the total duration of the key shots is controlled to be 15% of the original video duration. The selected key video shots are then spliced ​​together in the original time sequence to obtain the final video summary.

[0017] This invention also provides a large-model-assisted video summarization system based on global entity unification and robust importance scoring.

[0018] The present invention also provides a computer device.

[0019] The present invention also provides a computer-readable storage medium.

[0020] Compared with the prior art, the present invention has the following beneficial effects: This invention can effectively improve the accuracy and robustness of video summarization, while also enhancing video semantic understanding and keyframe recognition capabilities, thus promoting the application of video summarization algorithms in complex scenarios. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating the implementation process of the large model-assisted video summarization method based on global entity unification and robust importance scoring in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram illustrating the implementation process of the Global Entity Unification (GEU) module in this embodiment of the invention.

[0023] Figure 3 This is a schematic diagram illustrating the implementation process of the text cleaning module in an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram illustrating the implementation process of the multi-granularity importance scoring module in an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of the complete process of video summarization in an embodiment of the present invention. Detailed Implementation

[0026] The present invention will be further described in detail below through specific embodiments, but the embodiments of the present invention are not limited thereto.

[0027] The principle of this invention: This invention addresses two core problems in existing "video-text-video" paradigm video summarization methods: fragmented entity identities and unstable matching. Based on visual language models and large language models, it achieves a dual improvement in narrative coherence and key segment selection stability through the synergistic effect of Global Entity Unification (GEU) and Robust Importance Scoring (RIS) mechanisms. Its core principle lies in breaking the isolation of frame-level descriptions, establishing cross-frame entity semantic relationships, and eliminating the interference of text noise and semantic bias on frame selection through multi-granularity, two-level semantic matching and scoring. This allows the model to deeply understand the global narrative logic of the video while accurately capturing fine-grained visual information at the frame level. This invention reconstructs the semantic associations of frame-level descriptions using MapReduce logic: first, it generates frame-level descriptions with local entity labels through VLM; then, it aggregates fragmented entity information in the video time sequence, identifies the same physical entity that appears repeatedly in different frames and assigns a globally unified ID, transforming the originally independent frame-level texts into a coherent narrative description set with entity awareness. This allows the model to capture the long-term dynamics of entity relationships in the video, solving the narrative breakage problems caused by ambiguous entity references and missing cross-frame associations in traditional methods. To address the issue of unstable matching, this invention constructs a multi-level, multi-granular semantic matching system through a robust importance scoring mechanism: First, descriptive text cleaning is completed through visual-text cross-validation to filter out erroneous and redundant text noise, ensuring the accuracy of the input corpus. Then, a large model generates two-layer text summaries at the shot and video levels, taking into account both local shot details and the global theme of the video, bridging the semantic gap between single frames and the entire video. Finally, a multi-granular scoring strategy of fine-grained frame-shot scoring and coarse-grained shot-video scoring is adopted. The semantic gap between single-frame visual features and shot-level summaries is resolved through sentence weighted summation, and the relevance between shots and the global theme is measured through text semantic similarity. Finally, the two types of scores are fused to obtain a robust shot importance score, avoiding score fluctuations caused by minor changes in text wording, and ensuring that the selection of key shots is more in line with the true semantic focus of the video.

[0028] like Figure 1 , Figure 5 As shown in the figure, the present invention provides a large-model-assisted video summarization method based on global entity unification and robust importance scoring, which includes the following steps: (1) Frame sampling is performed on the input original video to obtain a discrete video frame sequence; a visual language model is used to generate entity-aware video descriptions for each sampled frame, generating frame-level local description text containing key entities and entity-identifiable attributes.

[0029] This step provides high-quality input for subsequent text summarization by performing refined preprocessing on the input video and generating entity-aware descriptive text. The quality of this stage directly affects the performance ceiling of subsequent models.

[0030] In one embodiment, extracting keyframes from the original video stream at a sampling rate of 3 frames per second achieves an optimal balance between maintaining the integrity of temporal information and computational efficiency in terms of temporal sampling. For a typical 60-second video, this sampling rate will produce 180 effective input frames, ensuring semantic continuity while controlling computational complexity.

[0031] In one embodiment, regarding entity-aware descriptive text generation, in the Map phase: Qwen3.5-Plus is used as the visual language model, with "Please describe this image in one short sentence, like person_a ..." as the generation prompt, and local entity labels (such as...) are generated for each sampled frame.<person_a> ,<car_b> The frame-level local description text of the generated description text contains key entities and their identifiable attributes (such as appearance, action, and location).

[0032] (2) The Map-Reduce logic is used to perform global entity unification processing on the frame-level local description text. In the Reduce stage: Qwen3.5-Plus is used to aggregate fragmented entity information (the same entity appearing at different times) in the video time sequence and identify the same entity that appears repeatedly in different frames. For example, the entity appearing at time point [0:08] is identified.<person_a> The time point [0:17] appeared<person_a> The time point [1:25] appeared<person_b> Determined to be the same entity and assigned standardization <entity> <index>A unified global ID in a standardized format is used to update the original frame-level local description text, forming a unified set of entity-aware description texts and eliminating the fragmentation of entity identities.

[0033] Both the Map and Reduce phases use Qwen3.5-Plus. In the Map phase, Qwen3.5-Plus is used to generate local description text for the frame. Then, in the Reduce phase, the frame and the local description text obtained in the Map phase are input into Qwen3.5-Plus, which integrates the description text for the same entity in the local description text.

[0034] (3) The image encoder using the CLIP model encodes all sampled frames of the unified frame-level description text, extracts the frame-level visual features, and uses the KTS algorithm to segment the original video into several non-overlapping video shots based on the frame-level visual features. Then, for the sampled frames in each shot, the cleaned description text of each frame is selected according to the cosine similarity formula, and description errors and redundant noise information are filtered out to obtain the cleaned description text set C′:

[0035] For the first The cleaned description text corresponding to the frame. For the first The set of descriptive text for the video shot containing the frame. For the first Visual features of a frame For describing text Textual features, , The function for calculating cosine similarity:

[0036] in, It is the dot product of vectors. and It is the length of the vector.

[0037] (4) A large model is used to generate two-layer text summaries at the shot level and video level from the cleaned descriptive text set, resulting in a shot-level summary text set and a video-level global summary text set.

[0038] In one embodiment, Tencent Hunyuan Big Model is used as the big language model to generate shot-level summaries for each shot according to instructions, resulting in shot-level summary text; then the shot-level summaries are concatenated to generate video-level global summary text, achieving a balance between shot-level details and video-level global themes.

[0039] Specifically, for the generation of shot-level summary text, the cleaned descriptive text of all sampled frames in each shot is concatenated into shot-level descriptive text, based on prompt words. "You are provided with a set of descriptions of a video. Please summarize what happened in a few sentences." This will generate a summary text that highly summarizes the semantic content of the video.

[0040] For the first A shot-level summary text for each shot. For large models, This is a text concatenation operation. For the first A collection of descriptive texts after the cleaning of each shot.

[0041] By iterating through all video shots, a set of shot-level summary texts is obtained. .

[0042] Lens-level summary text collection All shot-level summary texts are concatenated into video-level descriptive texts, using a large model and incorporating cue words. "The following is a set of descriptions corresponding to multiple shots in a video. Please summarize what happened in the video in a few paragraphs." (Generate video-level global summary text)

[0043] This is a global summary text at the video level. The total number of video shots after segmentation from the original video. This indicates the first shot-level summary text. This indicates the summary text for the second shot. Indicates the first A single-lens summary text.

[0044] (5) Multi-granularity importance scoring of frames. Combining frame-level visual features, the shot-level summary text generated in the above steps, and the video-level global summary text, multi-granularity importance scoring of fine-grained frame-shot scoring and coarse-grained shot-video scoring is calculated to obtain the frame-level importance score of each sampled frame and the shot-level importance score of each shot.

[0045] Fine-grained frame-to-shot scoring. The semantic similarity between a sampled frame and the shot-level summary of its corresponding shot is calculated. To compensate for the gap between the visual information of a single frame and the semantic information of the shot-level summary, a weighted sum of sentences in the shot-level summary is used to calculate the frame-to-shot score. Shot-level summary text for each shot Split into One sentence: A CLIP-based text encoder encodes each sentence to obtain a sentence feature set; a CLIP-based image encoder encodes all sampled frames within the shot to obtain a frame visual feature set. The weight of each sentence is calculated by evaluating its similarity to all sampled frames within the shot.

[0046] For the first The weight of each sentence, The first in this shot frame, This represents the total number of sampled frames within the shot. Sentences extracted by the text encoder of the CLIP model The higher the similarity between sentences, the greater their weight.

[0047] Then, based on sentence weight, the sentence weight within that shot is calculated. The frame-to-shot score is calculated by multiplying the similarity between the frame and each sentence by the weight of that sentence, and then calculating a weighted sum as the frame-to-shot score for that frame. The expression is as follows:

[0048] For frames Frame-shot rating. Iterate through all frames to obtain the frame-shot rating for all sampled frames.

[0049] Coarse-grained shot-to-video score is obtained by calculating the similarity between shot-level summary text and video-level summary text. The semantic similarity between the shot-level summary text and the global video-level summary text for each shot is calculated and used as the shot-to-video score for that shot.

[0050] For the first Shot-to-video rating for each shot. This provides the shot-level summary text features for this shot. These are text features for video-level global summarization.

[0051] Shot-level importance scores are fused. The frame-shot score of a frame and the shot-video score of its corresponding shot are weighted and summed to obtain the shot-level importance score for each video shot. :

[0052] These are weight parameters.

[0053] (6) Key shot selection and video summary generation. The 0 / 1 knapsack algorithm is used to select key video shots based on shot-level importance scores (or the highest number of shots with the highest shot-level importance scores can be selected directly, or the highest sum of several consecutive shots with the highest shot-level importance scores can be selected). The selected key video shots are spliced ​​together in the original time order to obtain the final video summary.

[0054] A shot-level importance score for each frame was obtained through multi-granularity importance scoring. In order to utilize Guide to the final video summary The generation of the summary video utilizes a 0-1 knapsack algorithm to select key segments. Specifically: First, calculate the shot-level importance score for all frames in each shot. The average value is used as the lens importance. ; Then, consider each shot as an element, the length of the segment As a factor of weight, the importance of the lens As the value of this element, the predefined summary video length As for the capacity of the backpack, the total length (i.e., total weight) of the selected lenses should not exceed [the limit]. Given the backpack capacity, select the key shots with the highest total importance (i.e., total value); After selecting the key shots, these segments are spliced ​​together in chronological order of the original video to obtain the final summary video. .

[0055] In one embodiment, the total duration of the key video shots is 15% of the original video duration.

[0056] The application and deployment of the method described in this embodiment of the invention supports multiple scenarios, including abnormal behavior detection in intelligent video surveillance, action understanding in human-computer interaction, and event localization in video content analysis. In particular, in medical rehabilitation training scenarios, the system can accurately identify whether a patient's rehabilitation movements are standardized, providing technical support for telemedicine.

[0057] In one embodiment, Table 1 shows the video summarization results of the method provided by the foregoing embodiments of the present invention on the SumMe dataset and the TVSum dataset.

[0058]

[0059] The method of this invention is compared with existing mainstream unsupervised video summarization methods (DRDSN, CSNet, RSGNu, LfVS, LGRLN, LS-CAN) on the SumMe and TVSum datasets. Performance is compared with Kendall's... And Spearman's Two rank correlation coefficients are used as evaluation indicators. The higher the index value, the higher the consistency between the importance score predicted by the model and the real score of the manual annotation, and the better the video summary effect. The value range is [-1,1], where -1 represents a perfect negative correlation and 1 represents a perfect positive correlation.

[0060] The test results are shown in Table 1. The method described in this embodiment of the invention implements Kendall's method on the SumMe dataset. =0.182, Spearman's =0.201, implementing Kendall's algorithm on the TVSum dataset. =0.196, Spearman's =0.223, significantly outperforming all existing comparative methods. Compared to the current best methods LGRLN (SumMe) and LS-CAN (TVSum), this invention achieves a higher SumMe score. Increase by 30% A 5.8% increase on TVSum An increase of 28.9%, The improvement of 17.4% fully verifies the effectiveness of the method of the present invention.

[0061] In one embodiment, Table 2 presents the ablation experiment results.

[0062] Table 2 Ablation Experiment Results

[0063] The ablation experiments show that: removing GEUs reduces model performance, proving that global entity unification can effectively solve the problem of identity fragmentation, enabling large models to better understand video content through descriptive text; removing descriptive text cleaning reduces performance, proving the importance of noise filtering in improving the quality of descriptive text, allowing large models to obtain more accurate video semantic information; removing frame-shot scoring results in the loss of fine-grained features and performance degradation; removing shot-video scoring results in the lack of global guidance, causing the model performance to almost collapse, fully verifying the necessity and synergistic effect of each core module of this invention.

[0064] The embodiments of this invention can effectively establish cross-frame entity semantic associations, eliminate the interference of text noise on frame selection, and enable the generated video summaries to have both coherent narrative logic and accurate core information. The performance on the SumMe and TVSum datasets is significantly better than existing unsupervised video summarization methods, and can be widely applied to scenarios such as video content analysis, intelligent video retrieval, short video generation, and intelligent monitoring.

[0065] In one embodiment, a large-model-assisted video summarization system based on global entity unification and robust importance scoring is provided to implement the method described in the foregoing embodiments. The system includes the following modules: The entity-aware video description generation module is used to sample the input raw video to obtain a discrete video frame sequence; and to generate an entity-aware video description for each sampled frame, generating frame-level local description text containing key entities and entity identifiable attributes. The global entity unification module is used to perform global entity unification processing on frame-level local description text, identify the same entity that appears repeatedly in different frames, assign a unified global ID to each entity, update the original frame-level local description text, and form a unified entity-aware description text set. The descriptive text cleaning module is used to clean the frame-level descriptive text after the global entity unified processing, extract frame-level visual features, segment the original video into several non-overlapping video shots based on the frame-level visual features, and then filter out the cleaned descriptive text of each frame to obtain the cleaned descriptive text set. The summary text generation module is used to generate two-level summary texts at the shot level and video level from the cleaned descriptive text set, resulting in shot-level summary text and video-level global summary text, respectively. The multi-granularity importance scoring module is used to combine frame-level visual features, shot-level summary text, and video-level global summary text to calculate multi-granularity importance scores of fine-grained frame-shot scores and coarse-grained shot-video scores. It takes the average of the frame-shot scores of all sampled frames in each video shot and adds it to the shot-video score of that shot to obtain the shot-level importance score of each video shot. The final video summary stitching module is used to select key video shots based on shot-level importance scores, and stitch the selected key video shots together in their original time order to obtain the final video summary.

[0066] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods described in the foregoing embodiments.

[0067] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described in the foregoing embodiments.

[0068] The system, device, and medium described herein have the same technical effects as those achieved by the methods described in the foregoing embodiments.

[0069] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.< / index> < / entity> < / index> < / entity>

Claims

1. A large-model-assisted video summarization method based on global entity unification and robust importance scoring, characterized in that, Includes the following steps: The input raw video is sampled frame by frame to obtain a discrete video frame sequence; entity-aware video description generation is performed on each sampled frame to generate frame-level local description text containing key entities and entity identifiable attributes; Global entity unification processing is performed on frame-level local description text to identify the same entity that appears repeatedly in different frames, assign a unified global ID to each entity, update the original frame-level local description text, and form a unified entity-aware description text set. The frame-level description text after global entity unified processing is cleaned, frame-level visual features are extracted, the original video is segmented into several non-overlapping video shots based on the frame-level visual features, and then the cleaned description text of each frame is selected to obtain the cleaned description text set. The cleaned descriptive text set is subjected to two-level summary text generation at the shot level and video level, resulting in shot-level summary text and video-level global summary text, respectively. By combining frame-level visual features, shot-level summary text, and video-level global summary text, multi-granularity importance scores are calculated, including fine-grained frame-shot scores and coarse-grained shot-video scores. The average frame-shot scores of all sampled frames within each video shot are then added to the shot-video score of that shot to obtain the shot-level importance score for each video shot. Key video shots are selected based on shot-level importance scores, and then spliced ​​together in their original chronological order to obtain the final video summary.

2. The large-model-assisted video summarization method based on global entity unification and robust importance scoring as described in claim 1, characterized in that, A visual language model is used to generate entity-aware video descriptions for each sampled frame.

3. The large-model-assisted video summarization method based on global entity unification and robust importance scoring as described in claim 1, characterized in that, A CLIP model image encoder is used to extract frame-level visual features, and the KTS algorithm is used to segment the original video.

4. The large-model-assisted video summarization method based on global entity unification and robust importance scoring as described in claim 1, characterized in that, The cleaned descriptive text for each frame is selected by using cosine similarity. Specifically, the descriptive text that is closest to the semantic content of the frame is selected by calculating the cosine similarity between the frame and the descriptive text, and this text is then used as the cleaned descriptive text.

5. The large-model-assisted video summarization method based on global entity unification and robust importance scoring as described in claim 1, characterized in that, The shot-level summary text is generated based on the shot-level description text, and the video-level summary text is generated based on the shot-level summary text.

6. The large-model-assisted video summarization method based on global entity unification and robust importance scoring as described in claim 1, characterized in that, The calculation steps for fine-grained frame-to-lens scoring are as follows: The shot-level summary text is split into several sentences, each sentence is encoded to obtain a sentence feature set, and all frames within the shot are encoded to obtain a frame visual feature set; The weight of each sentence is calculated by measuring its similarity to all sampled frames in the shot. Frame-shot scoring is calculated based on sentence weights; And / or, Coarse-grained shot-to-video scoring is obtained by calculating the similarity between shot-level and video-level summary texts.

7. A large-model-assisted video summarization method based on global entity unification and robust importance scoring according to any one of claims 1-6, characterized in that, The method is applicable to video content analysis, intelligent video retrieval, short video generation, or intelligent monitoring scenarios.

8. A large-model-assisted video summarization system based on global entity unification and robust importance scoring, characterized in that, The system for implementing the method according to any one of claims 1-7 includes the following modules: The entity-aware video description generation module is used to sample the input raw video to obtain a discrete video frame sequence; and to generate an entity-aware video description for each sampled frame, generating frame-level local description text containing key entities and entity identifiable attributes. The global entity unification module is used to perform global entity unification processing on frame-level local description text, identify the same entity that appears repeatedly in different frames, assign a unified global ID to each entity, update the original frame-level local description text, and form a unified entity-aware description text set. The descriptive text cleaning module is used to clean the frame-level descriptive text after the global entity unified processing, extract frame-level visual features, segment the original video into several non-overlapping video shots based on the frame-level visual features, and then filter out the cleaned descriptive text of each frame to obtain the cleaned descriptive text set. The summary text generation module is used to generate two-level summary texts at the shot level and video level from the cleaned descriptive text set, resulting in shot-level summary text and video-level global summary text, respectively. The multi-granularity importance scoring module is used to combine frame-level visual features, shot-level summary text, and video-level global summary text to calculate multi-granularity importance scores of fine-grained frame-shot scores and coarse-grained shot-video scores. It takes the average of the frame-shot scores of all sampled frames in each video shot and adds it to the shot-video score of that shot to obtain the shot-level importance score of each video shot. The final video summary stitching module is used to select key video shots based on shot-level importance scores, and stitch the selected key video shots together in their original time order to obtain the final video summary.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images