Video abstract generation method and related device

Abstract

The invention belongs to the technical field of video processing, and relates to a video abstract generation method and a related device. Performing frame sampling on the target video according to a preset target frame rate, and adjusting a sampling frame to a preset size to obtain a standardized frame sequence; a C3D feature extraction network is adopted, a sliding window is used for traversing the standardized frame sequence, convolution operation is carried out on the frame sequence in each window, and a fragment-level spatial-temporal feature sequence is output; and inputting the fragment-level spatial-temporal feature sequence into a C3D-Transform fusion abstract model, and outputting a final video abstract. The C3D-Transform fusion abstract model comprises an input layer, a position coding layer, a Transform encoder, a time sequence convolution layer, a normalization layer, a full connection layer and an output fragment importance score which are connected in sequence. The problems that feature modeling is limited, label construction is rough, an evaluation system is single, training efficiency is low and scene adaptability is poor in the prior art are solved.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, specifically relating to a video summarization method and related apparatus. Background Technology

[0002] With the widespread adoption of video recording equipment and the digitization of sports event broadcasting, sports video data has experienced explosive growth. These videos are typically long, containing numerous repetitive actions and redundant segments lacking key information, making it difficult for users to quickly grasp the core content of the event (such as key events like shots and scores). Video summarization technology has emerged to address this need. Its core principle is to automatically analyze the spatiotemporal information of videos, extract key segments, and condense them into short videos, helping users efficiently grasp the core content of the videos.

[0003] Currently, video summarization technology is mainly divided into traditional non-learning methods (such as clustering and keyframe extraction) and deep learning methods (such as models based on CNN, RNN, and Transformer). Among them, deep learning methods have become the mainstream research direction due to their powerful feature extraction and temporal dependency modeling capabilities. In the field of sports videos, existing methods are mostly based on single-modal visual features (such as frame-level image features), predicting the importance score of segments through models, and then selecting key segments according to a fixed ratio to generate summaries.

[0004] Defects and shortcomings of existing technology: Coarse label construction: Existing technologies mostly use binary hard labels (0 represents non-critical segments, 1 represents critical segments), which cannot quantify the contribution of segments to critical events, resulting in unstable gradients during model training and insufficient accuracy in capturing critical events.

[0005] Limitations of feature modeling: Single visual features are difficult to fully represent key event information in sports videos, and traditional models (CNN / RNN) are difficult to effectively capture long temporal dependencies, while pure Transformer models lack the ability to model local spatiotemporal features and easily ignore the correlation between adjacent segments.

[0006] The loss function design is too simplistic: Existing methods mostly rely on a single mean squared error (MSE) loss, which only focuses on the numerical fit between the predicted score and the label, without strengthening the ranking difference between key and non-key segments, resulting in insufficient rationality in the selection of summary segments.

[0007] The evaluation system is inadequate: existing evaluations mostly focus on score fit (such as F1 scores and correlation coefficients), lacking specific evaluations for key events in sports videos, and failing to accurately measure the summaries' coverage of core event content.

[0008] The summary generation lacks flexibility: the summary length control method in the reasoning stage is rigid and does not take into account the prior information of events in sports videos, making it difficult to adapt to the differentiated needs of different scenarios. Summary of the Invention

[0009] The purpose of this invention is to provide a video summarization method and related apparatus, which solves the problems of limited feature modeling, coarse label construction, single evaluation system, low training efficiency and poor scene adaptability in the prior art.

[0010] This invention is achieved through the following technical solution: A video summarization method includes the following steps: S1. Sample the target video at a preset target frame rate, and adjust the sampled frames to a preset size to obtain a standardized frame sequence. S2. Using the C3D feature extraction network, the standardized frame sequence is traversed through a sliding window. Convolution operation is performed on the frame sequence in each window to output a fragment-level spatiotemporal feature sequence. S3. Input the fragment-level spatiotemporal feature sequence into the trained C3D-Transformer fusion summarization model and output the fragment importance score; The C3D-Transformer fusion summarization model includes an input layer, a positional encoding layer, a Transformer encoder, a temporal convolutional layer, a normalization layer, and a fully connected layer connected in sequence, and outputs a fragment importance score. S4. Obtain keyframes based on segment importance scores, and combine the keyframes to output the final video summary.

[0011] Furthermore, in S2, the C3D feature extraction network is a pre-trained 3D convolutional neural network, containing multiple 3D convolutional layers and 3D pooling layers, denoted as model. .

[0012] Furthermore, in S2, the step of traversing the standardized frame sequence using a sliding window, performing convolution operations on the frame sequence within each window, and outputting a segment-level spatiotemporal feature sequence is specifically as follows: Suppose the sliding window contains Frame, window step size is Traverse the standardized frame sequence of a single video. The frame sequence for each window is ,in, T is the total number of segments; For the first i Each frame is a standardized image; N This refers to the total number of frames in the single video. Through the model Extracting the first The spatiotemporal characteristics of each segment: ; The final output is a fragment-level spatiotemporal feature sequence: , This represents the spatiotemporal characteristics of the T-th segment.

[0013] Furthermore, in S3, the position encoding layer is used to capture timing information and add sinusoidal position encoding. The encoding formula is: ; Encoded features are ; P represents the input segment-level spatiotemporal feature sequence, and P is the sinusoidal position encoding matrix added to the features, used to capture the temporal position information of the segment; The temporal convolutional layer is a 1D convolutional layer with convolutional kernels. bias The output is: ; in, The transpose of E ( ), transpose after convolution T represents the total number of fragments; B represents the batch size. Normalized layer: To enhance model stability; The fully connected layer maps the 512-dimensional features to a 1-dimensional importance score. ,in, , Output ; These are the features of C after layer normalization; It is the weight parameter matrix of the fully connected layer.

[0014] Furthermore, in S3, the Transformer encoder contains N encoder layers. The output of each encoder layer serves as the input of the next encoder layer. After completing the N layers of calculations in sequence, the final encoder feature is output. Each encoder layer consists of a multi-head self-attention layer, a first residual connection and normalization layer, a feedforward network layer, and a second residual connection and normalization layer, connected in sequence.

[0015] Furthermore, the loss function of the C3D-Transformer fusion summarization model is a multi-objective loss function, consisting of mask mean square error loss, pairwise ranking loss, local ranking loss, and orientation anchoring loss, with the specific formula as follows: ; in , , These are the weighting coefficients; The loss is for pairwise sorting; This is the mask mean square error loss; This is the local sorting loss; Loss due to directional anchoring.

[0016] Furthermore, in S4, the process of obtaining keyframes based on fragment importance scores specifically involves: Normalize the fragment importance scores using the sigmoid function to obtain the normalized result; the expression is: ; Represents the importance score of each segment. Represents the normalization result; Based on the normalization results, segment selection is performed under the basic strategy, or segment selection is performed under the overlay mode of the basic strategy and the event-aware strategy to obtain keyframes; The basic strategy is: according to Sort in descending order and select the first few. There are 10 segments; where r is the proportion of the abstract length and T is the total number of segments. The overlay mode of basic strategy and event-aware strategy is as follows: First press Sort in descending order and select the first few. 0 segments; Event-aware strategy: Define the prior interval of the event. ,calculate The center time of each segment in the segment Does it fall within the prior interval of the event? If so, then the segment belongs to the key event-related segment; judge in turn until the previous segment is selected. A segment; in, Greater than , Represents the start timestamp of the m-th critical event; Represents the end timestamp of the m-th critical event; k is any number from 1 to m.

[0017] This invention discloses a video summarization generation system, comprising: The preprocessing module is used to sample the target video at a preset target frame rate, and adjust the sampled frames to a preset size to obtain a standardized frame sequence. The feature extraction module is used to employ the C3D feature extraction network to traverse the standardized frame sequence with a sliding window, perform convolution operations on the frame sequence within each window, and output a fragment-level spatiotemporal feature sequence. The prediction module is used to input fragment-level spatiotemporal feature sequences into a trained C3D-Transformer fusion summarization model and output fragment importance scores. The C3D-Transformer fusion summarization model includes an input layer, a positional encoding layer, a Transformer encoder, a temporal convolutional layer, a normalization layer, and a fully connected layer connected in sequence, and outputs a fragment importance score. The output module is used to obtain keyframes based on the importance score of the segments, and combine the keyframes to output the final video summary.

[0018] The present invention discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video summarization generation method.

[0019] The present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video summarization generation method.

[0020] Compared with the prior art, the present invention has the following beneficial technical effects: This invention discloses a video summarization method. By sampling and adjusting the preset frame rate and size, input videos of arbitrary length and resolution are unified into structured data, significantly reducing the complexity of subsequent processing and enhancing the method's adaptability and robustness to different source videos. A C3D network is used to extract features in a sliding window manner, simultaneously capturing spatial appearance information and short-term temporal motion information within video segments, generating semantically rich segment-level spatiotemporal feature sequences, laying a precise data foundation for importance assessment. The segment-level spatiotemporal feature sequences are input into a trained C3D-Transformer fusion summarization model. The C3D-Transformer fusion summarization model introduces a Transformer encoder, whose core self-attention mechanism effectively models global dependencies across long distances between video segments, enabling the model to evaluate the importance of individual segments based on the overall context of the video, avoiding local optima. The model structure innovatively combines the local spatiotemporal features of C3D, the global contextual association of Transformer, and the extraction of local coherence patterns from temporal convolutional layers. This multi-layered fusion architecture of "local-global-local" enables the model to grasp both the macro-narrative structure and the micro-action connections, thereby generating summaries that are more consistent with cognitive logic. In summary, this method, through standardized preprocessing, deep integration of C3D and Transformer, and multi-layered temporal modeling, achieves efficient and accurate automated summarization of video content, demonstrating significant technical improvements in the keyness, coherence, and completeness of the summaries.

[0021] Furthermore, the Transformer encoder captures long temporal associations through multi-head self-attention, and the temporal convolutional layer enhances local feature modeling, solving the problem of the lack of local inductive bias in the pure Transformer, making the segment importance score more accurate (soft F1 score can reach above 0.74).

[0022] Furthermore, the loss function of the C3D-Transformer fusion summarization model is a multi-objective loss function, consisting of mask mean square error loss, pairwise ranking loss, local ranking loss, and orientation anchoring loss. Pairwise ranking loss and local ranking loss work together to enhance the difference between key segments and non-key segments, while orientation anchoring loss ensures the saliency of key segments. This solves the problem of insufficient ranking ability of a single MSE loss, making the summary segment selection more in line with user needs.

[0023] The event-level assessment program targets key events in sports videos, overcoming the shortcomings of existing assessments that only focus on score fitting. It can more comprehensively measure summary quality and provide a dedicated assessment standard for sports video scenarios.

[0024] It supports two modes: basic ratio filtering and event-aware filtering. The summary length can be flexibly adjusted through the ratio parameter to adapt to different application scenarios (such as short video platforms and event review), making it more universal.

[0025] The system enables end-to-end video summarization through a web service. It is easy to operate, highly efficient, supports batch processing and multi-hardware environment adaptation, and has practical application value.

[0026] The feature encoding method based on the C3D-Transformer fusion architecture improves the accuracy of segment importance scoring by extracting spatiotemporal features through 3D convolution, capturing long-term temporal correlations through Transformer, and enhancing local spatiotemporal features through 1D convolution.

[0027] A two-dimensional video summarization evaluation system combining basic and event-level metrics, including proprietary calculation methods for event recall and event precision, can quantify the coverage effect of key events. Attached Figure Description

[0028] Figure 1 This is an overall flowchart of a video summarization method according to the present invention; Figure 2 This is a diagram of the C3D-Transformer fusion summary model architecture. Figure 3 This is a flowchart of the training process for the C3D-Transformer fusion summarization model. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the present invention, and not all of them.

[0030] The detailed description of the embodiments of the present invention provided in the following figures is not intended to limit the scope of the claimed invention, but merely to illustrate one selected embodiment of the invention. All other embodiments obtained by those skilled in the art based on the figures and embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0031] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0032] This invention provides a video summarization generation method based on multi-module collaborative optimization, such as... Figure 1 As shown, it includes video preprocessing, feature extraction, soft real label construction, model training, summary generation, and two-dimensional evaluation.

[0033] I. Video Preprocessing Input: Sports video file (e.g., basketball game .mp4 format), configuration parameters (target frame rate TARGET_FPS, segment length) Sliding step size ).

[0034] The processing flow is as follows: 1. Video frame sampling: Read the video using OpenCV, assuming the original video frame rate is... The target frame rate is Then the sampling interval The frame sequence is sampled at intervals I to ensure that timing information is not lost.

[0035] 2. Frame Preprocessing: Resize the sampled frames to 112×112 pixels to unify the image size. Denote a single frame as... .

[0036] II. Feature Extraction from Sliding Window Load the C3D feature extraction network, which is a pre-trained 3D convolutional neural network containing multiple 3D convolutional layers and 3D pooling layers; remove the fully connected layers of the C3D feature extraction network, retaining only the convolutional and pooling layers used to extract spatiotemporal features, and label the retained feature extraction part as the model. .

[0037] Suppose the sliding window contains Frame, window step size is Traverse the normalized frame sequence The frame sequence for each window is ,in , T The total number of segments; L The number of frames within the window; N Represents the total number of frames in a single video; For the first i Frame-normalized image.

[0038] Feature extraction using a model: The final output is a spatiotemporal feature sequence: , Representing the T The spatiotemporal characteristics of each segment.

[0039] Feature saving: The spatiotemporal feature sequence F is stored as a .npy file according to the video ID, which is the video ID matching feature file, and is used for subsequent model training.

[0040] III. Construction of Soft Authentic Labels Input: Video key event annotation data Spatiotemporal feature sequence F, configuration parameters (Gaussian kernel scaling factor) (Power coefficient p).

[0041] in, These are the start and end times of the kth critical event, respectively; the video critical event annotation data is known information obtained in advance through manual annotation (such as the LabelStudio tool).

[0042] Processing flow: 1. Annotation parsing: Read the event annotation file and parse it to obtain the set of key event time intervals E for each video.

[0043] 2. Segment Time Mapping: Let the duration of a single frame be... Then the start time of the t-th segment End time Segment center time .

[0044] 3. Gaussian kernel score calculation: for each key event Event Center Event duration Gaussian kernel standard deviation The contribution of segment t to event k is a Gaussian function: The total contribution of fragment t is the sum of the contribution values ​​of all events: .

[0045] 4. Label Normalization: The total contribution value is linearly normalized to obtain the initial soft labels. By adjusting the label distribution using power coefficients, the final soft true labels are: (p is the power coefficient, default is 1.0), label sequence Save it as a .npy file, which is the soft tag file.

[0046] IV. Integrated Model Input Each sports video file corresponds to a video ID matching feature file and a soft label file. A large number of sports video files and their corresponding video ID matching feature files and soft label files constitute a dataset.

[0047] Data loading: Match feature files and soft tag files by video ID to read segment-level visual features. Tag sequence and sequence length ,in The value is equal to the total number of video clips T, only the scene descriptions are different.

[0048] Data alignment: Padding is performed on feature sequences that are too short. Specifically, the maximum sequence length is set to... ,right The sequence is used to fill in the features of the last frame: Labels are filled with 0 to ensure consistent data dimensions during batch training.

[0049] The integrated format is dictionary-type data, containing visual features ( B represents the batch size, and the importance score is ( ). ), sequence length ( ) and video ID.

[0050] V. Model Architecture Design The core model of this invention is C3D-Transformer, which integrates Transformer encoder and temporal convolutional network. The architecture is shown in Figure 2. The input layer is fragment-level visual features, which pass through the position encoding layer, Transformer encoder, temporal convolutional layer, normalization layer, and fully connected layer in sequence, and output fragment importance score.

[0051] Input layer: Batch video clip features (B is the batch size, T is the sequence length).

[0052] Position coding layer: Adds sinusoidal position coding to capture timing information. The encoding formula is: The encoded features are .

[0053] like Figure 2 As shown, the Transformer encoder contains M encoder layers (M=2 by default), and the structure of each encoder layer is as follows: 1. Multi-head self-attention layer (nhead=8): Maps input features to query Q, key K, and value V, all with the same dimension. It is divided into 8 attention heads, each with a dimension of 1. The output of a single attention head is: Concatenate all header outputs and perform a linear transformation: ; in, These are learnable parameters. These represent the query, key, and value vectors corresponding to the h-th attention head, respectively, and are used to calculate the association weight of that attention head.

[0054] 2. Residual Connectivity and Normalization Layer: .

[0055] 3. Feedforward Network Layer (FFN): Contains two layers of linear transformation and ReLU activation. ,in , .

[0056] 4. Residual Connectivity and Normalization Layer: Output encoder features .

[0057] Temporal convolutional layer: 1D convolutional layer (kernel_size=3, padding=1, stride=1), convolutional kernel bias The output is: in The transpose of E ( ), transpose after convolution .

[0058] Where kernel_size represents the kernel length; padding represents the padding length at both ends of the sequence; and stride represents the kernel stride.

[0059] Normalized layer: To enhance model stability, the layer normalization method employed first normalizes the dimensions of the input features, stabilizing the mean and variance of each layer's input and preventing gradient vanishing or exploding during training. Then, by standardizing the feature scale, gradient updates during model training become smoother, improving convergence speed and training efficiency. Finally, it reduces internal covariate bias, lowering the model's sensitivity to changes in input data distribution and improving its generalization performance across different scenarios. This represents the output of this layer.

[0060] Fully connected layer: maps 512-dimensional features to 1-dimensional importance scores. ,in , Output (Sigmoid activation is not added during the training phase, but is added during the inference phase).

[0061] VI. Multi-objective training loss system Design a multi-loss fusion strategy to balance numerical fitting and ranking rationality; the training process is as follows: Figure 3 As shown, the steps include data loading, forward propagation, multiple loss calculation, gradient backpropagation, and model saving: Core loss function: 1. Mask mean square error: Only calculates the loss within the effective sequence length. Let the effective length of the i-th sample in the batch be... The predicted score is The real label is The loss formula is: .

[0062] in, Let represent the predicted score of the t-th segment of the i-th sample. B represents the true label of the t-th segment of the i-th sample; B is the batch size.

[0063] 2. Pairwise ranking loss: Construct "key segment - non-key segment" sample pairs. Let the set of key segments in the i-th sample be... The difference threshold, This is a set of non-critical fragments; The loss formula is: ; in, The total number of sample pairs; , where is the margin value, is a hyperparameter that controls the boundary threshold between positive and negative samples; Let be the true label of the s-th segment of the i-th sample; The model predicts the label for the s-th segment of the i-th sample.

[0064] 3. Local sorting loss: Sorting is constrained within a local window (window size w=5), and the loss formula is:

[0065] in, This is an indicator function (1 if the condition is met, 0 otherwise).

[0066] 4. Directional Anchoring Loss: Force key segment scores above a threshold Let the set of key segments be... The loss formula is:

[0067] Loss fusion: The total loss is the weighted sum of the individual losses, as shown in the formula: ; in, , , The weights are calculated using the Adam optimizer (learning rate). Weight decay Minimize the total loss and update the model parameters.

[0068] Training aids: Regularly save the optimal model (based on validation set loss) and visualize the importance curve, score distribution, and summary fragment selection results to assist in monitoring model performance.

[0069] The training process is a closed-loop optimization process consisting of 30 iterations: first, based on the batch size... B Load the feature and soft-label dataset and pad and align the sequences; then input the batch features into the model and perform forward propagation to obtain the predicted scores of each segment; next, combine the true labels and effective length to calculate the total loss consisting of pairwise sorting loss, mask mean square error, local sorting loss, and orientation anchoring loss; then update the model parameters using the Adam optimizer through gradient backpropagation; finally, when the validation set loss reaches its optimum, save the model weights to complete the training.

[0070] After training, the model is evaluated using the following basic evaluation metrics: 1. Soft F1 score: Among them, accuracy Recall rate .

[0071] 2. Coverage: .

[0072] Event-level evaluation metrics: deducing key event intervals from soft reality labels Let the time interval for generating the summary be . ,but: Event recall rate: .

[0073] Event accuracy: .

[0074] Event F1 score: .

[0075] The training results of the C3D-Transformer fusion summarization model are shown in Table 1 below.

[0076] Table 1

[0077] The multi-dimensional evaluation system of this invention not only includes the overall F1 score that measures the quality of the global summary, but also sets targeted indicators such as Event-F1-Score, event recall, and coverage. This rigorous design clearly shows that although the overall F1 score is slightly lower, this is a reasonable trade-off made by the model to prioritize the core event recognition capability. With an excellent performance of 75% Event-F1-Score, 90% event recall, and 65% coverage, it is fully demonstrated that this model can accurately and comprehensively capture key events in event-intensive scenarios such as sports videos, effectively meeting users' needs for obtaining core content. Its outstanding advantage in event recognition capability has thus been more clearly verified.

[0078] VII. Practical Applications Input: Sports video to be processed, trained model, summary length ratio r (e.g., 15%, 25%), event awareness switch.

[0079] Processing flow: 1. Feature Extraction: Repeat the "Preprocessing and Feature Extraction" steps for the video to be processed to obtain segment-level spatiotemporal feature sequences. .

[0080] 2. Score Prediction: Load the optimal model and input... Output the importance score of each segment. Normalization is achieved using the sigmoid function: ; This is The transpose of .

[0081] 3. Segment Filtering: The first approach is a basic strategy: [based on...] Sort in descending order and select the first few. A fragment.

[0082] The second method: First press Sort in descending order and select the first few. 0 segments; Event-aware strategy: Define the prior interval of the event. ,calculate The center time of each segment in the segment Does it fall within the prior interval of the event? If yes, then the segment belongs to the segment related to the key event; otherwise, it does not. This process is repeated until a segment is selected. This segment enhances the coverage of key events.

[0083] in, Greater than , Represents the start timestamp of the m-th critical event; Represents the end timestamp of the m-th critical event; k is any number from 1 to m.

[0084] The basic strategy must be executed, and the event-aware strategy is an enhancement on the basic strategy. It can be a single basic strategy or a superposition mode of "basic strategy + event-aware strategy".

[0085] 4. Summary generation: The selected segments, i.e. keyframes, are spliced ​​together in their original time sequence to generate the final video summary.

[0086] Theoretically, this invention can be extended to videos with a time-series distribution of key events, with basketball and football being typical examples in sports.

[0087] This invention discloses a video summarization generation system, comprising: The preprocessing module is used to sample the target video at a preset target frame rate, and adjust the sampled frames to a preset size to obtain a standardized frame sequence. The feature extraction module is used to employ the C3D feature extraction network to traverse the standardized frame sequence with a sliding window, perform convolution operations on the frame sequence within each window, and output a fragment-level spatiotemporal feature sequence. The prediction module is used to input fragment-level spatiotemporal feature sequences into a trained C3D-Transformer fusion summarization model and output fragment importance scores. The C3D-Transformer fusion summarization model includes an input layer, a positional encoding layer, a Transformer encoder, a temporal convolutional layer, a normalization layer, and a fully connected layer connected in sequence, and outputs a fragment importance score. The output module is used to obtain keyframes based on the importance score of the segments, and combine the keyframes to output the final video summary.

[0088] This invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the video summarization generation method. The memory may include main memory, such as high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device. The processor, network interface, and memory are interconnected via an internal bus, which may be an industry-standard architecture bus, a peripheral component interconnection standard bus, an extended industry-standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store the program; specifically, the program may include program code, which includes computer operation instructions. The memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0089] This invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the video summarization method. Specifically, the computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include random access memory and / or cache memory, etc. The non-volatile memory may include read-only memory, hard disk, flash memory, optical disk, magnetic disk, etc.

[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A video summarization method, characterized in that, Includes the following steps: S1. Sample the target video at a preset target frame rate, and adjust the sampled frames to a preset size to obtain a standardized frame sequence. S2. Using the C3D feature extraction network, the standardized frame sequence is traversed through a sliding window. Convolution operation is performed on the frame sequence in each window to output a fragment-level spatiotemporal feature sequence. S3. Input the fragment-level spatiotemporal feature sequence into the trained C3D-Transformer fusion summarization model and output the fragment importance score; The C3D-Transformer fusion summarization model includes an input layer, a positional encoding layer, a Transformer encoder, a temporal convolutional layer, a normalization layer, and a fully connected layer connected in sequence, and outputs a fragment importance score. S4. Obtain keyframes based on segment importance scores, and combine the keyframes to output the final video summary.

2. The video summarization method according to claim 1, characterized in that, In S2, the C3D feature extraction network is a pre-trained 3D convolutional neural network, containing multiple 3D convolutional layers and 3D pooling layers, denoted as model. .

3. The video summarization method according to claim 2, characterized in that, In S2, the step of traversing the standardized frame sequence using a sliding window, performing convolution operations on the frame sequence within each window, and outputting a segment-level spatiotemporal feature sequence is specifically as follows: Suppose the sliding window contains Frame, window step size is Traverse the standardized frame sequence of a single video. The frame sequence for each window is ,in, T is the total number of segments; For the first i Each frame is a standardized image; N This refers to the total number of frames in the single video. Through the model Extracting the first The spatiotemporal characteristics of each segment: ; The final output is a fragment-level spatiotemporal feature sequence: , This represents the spatiotemporal characteristics of the T-th segment.

4. The video summarization method according to claim 1, characterized in that, In S3, the position coding layer is used to capture timing information and add sinusoidal position coding. The encoding formula is: ; Encoded features are ; P represents the input segment-level spatiotemporal feature sequence, and P is the sinusoidal position encoding matrix added to the features, used to capture the temporal position information of the segment; The temporal convolutional layer is a 1D convolutional layer with convolutional kernels. bias The output is: ; in, The transpose of E ( ), transpose after convolution T represents the total number of fragments; B represents the batch size. Normalized layer: To enhance model stability; The fully connected layer maps the 512-dimensional features to a 1-dimensional importance score. ,in, , Output ; These are the features of C after layer normalization; It is the weight parameter matrix of the fully connected layer.

5. The video summarization method according to claim 1, characterized in that, In S3, the Transformer encoder contains N encoder layers. The output of each encoder layer is used as the input of the next encoder layer. After completing the N layers of calculation, the final encoder feature is output. Each encoder layer consists of a multi-head self-attention layer, a first residual connection and normalization layer, a feedforward network layer, and a second residual connection and normalization layer, connected in sequence.

6. The video summarization method according to claim 1, characterized in that, The loss function of the C3D-Transformer fusion summarization model is a multi-objective loss function, consisting of mask mean square error loss, pairwise ranking loss, local ranking loss, and orientation anchoring loss. The specific formula is as follows: ； in , , These are the weighting coefficients; The loss is for pairwise sorting; This is the mask mean square error loss; This is the local sorting loss; Loss due to directional anchoring.

7. The video summarization method according to claim 1, characterized in that, In S4, obtaining keyframes based on fragment importance scores specifically involves: Normalize the fragment importance scores using the sigmoid function to obtain the normalized result; the expression is: ; Represents the importance score of each segment. Represents the normalization result; Based on the normalization results, segment selection is performed under the basic strategy, or segment selection is performed under the overlay mode of the basic strategy and the event-aware strategy to obtain keyframes; The basic strategy is: according to Sort in descending order and select the first few. There are 10 segments; where r is the proportion of the abstract length and T is the total number of segments. The overlay mode of basic strategy and event-aware strategy is as follows: First press Sort in descending order and select the first few. 0 segments; Event-aware strategy: Define the prior interval of the event. ,calculate The center time of each segment in the segment Does it fall within the prior interval of the event? If so, then the segment belongs to the key event-related segment; judge in turn until the previous segment is selected. A segment; in, Greater than , Represents the start timestamp of the m-th critical event; Represents the end timestamp of the m-th critical event; k is any number from 1 to m.

8. A video summarization system, characterized in that, include: The preprocessing module is used to sample the target video at a preset target frame rate, and adjust the sampled frames to a preset size to obtain a standardized frame sequence. The feature extraction module is used to employ the C3D feature extraction network to traverse the standardized frame sequence with a sliding window, perform convolution operations on the frame sequence within each window, and output a fragment-level spatiotemporal feature sequence. The prediction module is used to input fragment-level spatiotemporal feature sequences into a trained C3D-Transformer fusion summarization model and output fragment importance scores. The C3D-Transformer fusion summarization model includes an input layer, a positional encoding layer, a Transformer encoder, a temporal convolutional layer, a normalization layer, and a fully connected layer connected in sequence, and outputs a fragment importance score. The output module is used to obtain keyframes based on the importance score of the segments, and combine the keyframes to output the final video summary.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video summarization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the video summarization method as described in any one of claims 1 to 7.

Images