Video summary generation method and system based on feature expression enhancement

Abstract

The present invention discloses a method and system for generating video summaries based on feature expression enhancement, which relates to the field of computer vision technology. The method comprises the following steps: obtaining an original video and extracting a video frame image sequence; inputting the video frame image sequence into an OCVF video understanding model; firstly extracting initial features from each input video frame image through a feature extraction module; extracting key spatiotemporal features from the initial features through an attention module; then generating a high-level feature representation through feature mapping and skip connections; and finally outputting the importance score of each video frame through a classifier to determine key frames and generate a final video summary result; wherein the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer, and a spatiotemporal attention layer arranged in sequence. The present invention can achieve more comprehensive modeling and efficient understanding of video content, and realize more robust and accurate video summary generation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for generating a video summary based on feature expression enhancement. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Video summarization is a key task in the field of natural language processing (NLP) and a current research hotspot. With the rapid development of short video platforms, video data is experiencing explosive growth. Efficiently understanding and extracting key information from videos, and then generating concise and accurate video summaries, is a core challenge that needs to be addressed. Video summarization involves extracting key information from long videos and compressing them into a short, easily digestible format, allowing users to quickly grasp the core content.

[0004] Currently, video summarization focuses on extracting key information, particularly summarizing the core content of a video through keyframes. However, this approach often has numerous shortcomings. For example, existing research typically analyzes key features of video frames to compress video information while retaining important elements. However, the extracted features are often not representative enough. With the development of neural networks and the powerful feature extraction capabilities of CNNs (Convolutional Neural Networks), CNNs are often used to extract and analyze video frame features in depth, thereby improving the representativeness of video frame features. However, this approach does not fully utilize the spatial location information in the video, resulting in incomplete feature extraction and poor accuracy of the generated video summary.

[0005] Furthermore, relying solely on convolutional neural networks to extract features from video frames still struggles to fully capture complex spatiotemporal relationships. To enhance the modeling of spatiotemporal features in videos, current approaches are introducing temporal convolution or attention mechanisms to enhance this capability, combining the extracted features to provide a more comprehensive understanding of the video. While these improvements have achieved promising results in specific scenarios, they still have certain shortcomings, such as an inability to fully capture key features in long-term videos and insufficiently refined feature representation of different modalities. Summary of the Invention

[0006] To address the shortcomings of the above-mentioned existing technologies, the present invention provides a method and system for generating video summaries based on enhanced feature expression. This method integrates advanced temporal modeling and attention strategies to construct a new video understanding model, the Optimized Channel Value Framework (OCVF). This OCVF model is used to achieve more comprehensive modeling and efficient understanding of video content, thereby achieving more robust and accurate video summary generation.

[0007] In a first aspect, the present invention provides a method for generating a video summary based on feature expression enhancement.

[0008] A video summary generation method based on feature expression enhancement, comprising:

[0009] Get the original video and extract the video frame image sequence;

[0010] The video frame image sequence is input into the OCVF video understanding model. Each input video frame image is first extracted by the feature extraction module to extract the initial features. The initial features are then used by the attention module to extract key spatiotemporal features. Then, high-level feature representations are generated through feature mapping and skip connections. Finally, the high-level feature representations output the importance score of each video frame through the classifier, determine the key frames, and generate the final video summary result.

[0011] Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer arranged in sequence; the initial features are adaptively adjusted to the channel weights by the channel attention layer to generate key features; after the dimensionality reduction processing by the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted by the self-attention layer and the spatiotemporal attention layer.

[0012] In a further technical solution, in the channel attention layer, the input initial feature map is subjected to global average pooling to compress the spatial information to obtain a global description of each channel, and then the channel attention information is learned through convolution, and the activation function is used to obtain the attention weight of each channel;

[0013] Based on the attention weight of each channel, the original initial feature map of the input is multiplied element by element to obtain the channel-weighted feature map, that is, the key feature map.

[0014] A further technical solution is that in the position coding layer, for the feature representation of each input video frame image, the position information of the current video frame in the original video is introduced to obtain the feature representation after position coding;

[0015] The position information is represented by a position coding matrix, which is calculated using any one or more methods of fixed position coding, relative position coding, and learned position coding.

[0016] A further technical solution is to extract key spatiotemporal features from the position-encoded features through the self-attention layer and the spatiotemporal attention layer in sequence, including:

[0017] The position-encoded features are input into the self-attention layer as input features. The attention weight matrix is ​​calculated through the self-attention mechanism, and the updated feature representation is obtained by weighted summation of the attention weight matrix.

[0018] The updated feature representation is then input into the spatiotemporal attention layer as input features. The temporal and spatial attention weight matrices are calculated through the temporal and spatial attention mechanisms respectively. Based on the two attention weight matrices and using a dynamic weight adjustment mechanism, feature weighted fusion is performed to obtain the key spatiotemporal features.

[0019] A further technical solution is that the dynamic weight adjustment mechanism is:

[0020] The scaling factor is calculated based on the number of time frames and spatial dimensions of the original video;

[0021] According to the scaling factor, the weights of feature weighted fusion are adaptively adjusted to balance the importance of time and space.

[0022] In a further technical solution, the feature map and the jump connection are:

[0023] Use multi-layer perceptron to perform nonlinear mapping on input features to generate complex feature representations;

[0024] Through skip connections, the complex feature representation is fused with the extracted initial features, multi-level intermediate features, and key spatiotemporal features to obtain fused features; among them, the multi-level intermediate features include key features;

[0025] The fused features are then subjected to multiple nonlinear mappings to generate the final embedded representation, i.e., the high-level feature representation.

[0026] In a second aspect, the present invention provides a video summary generation system based on feature expression enhancement.

[0027] A video summary generation system based on feature expression enhancement, comprising:

[0028] Video acquisition module, used to acquire original video and extract video frame image sequence;

[0029] The video summary generation module is used to input the video frame image sequence into the OCVF video understanding model. Each input video frame image is first extracted by the feature extraction module to extract the initial features. The initial features are then used by the attention module to extract key spatiotemporal features. Then, high-level feature representations are generated through feature mapping and skip connections. Finally, the high-level feature representations output the importance score of each video frame through the classifier, determine the key frames, and generate the final video summary result.

[0030] Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer arranged in sequence; the initial features are adaptively adjusted to the channel weights by the channel attention layer to generate key features; after the dimensionality reduction processing by the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted by the self-attention layer and the spatiotemporal attention layer.

[0031] In a third aspect, the present invention further provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned method for generating a video summary based on feature expression enhancement when executing the executable instructions stored in the memory.

[0032] In a fourth aspect, the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned method for generating a video summary based on feature expression enhancement.

[0033] In a fifth aspect, the present invention also provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned method for generating a video summary based on feature expression enhancement is implemented.

[0034] One or more of the above technical solutions have the following beneficial effects:

[0035] 1. The present invention provides a method and system for generating video summaries based on enhanced feature expression, integrating advanced temporal modeling and attention strategies to construct an OCVF video understanding model. The OCVF model is used to achieve more comprehensive modeling and efficient understanding of video content, thereby achieving more robust and accurate video summary generation.

[0036] 2. In the OCVF model proposed in this invention, by introducing the channel attention mechanism, the importance of each feature channel can be adaptively adjusted to highlight the key features, thereby effectively concentrating the features that are helpful for the task, improving the effect of video understanding, and enhancing the module's ability to express video features; through temporal modeling, it can help the model better understand the temporal relationship of frame sequences and enhance the modeling ability of global context; then, through the self-attention mechanism, the global dependency between video frames is modeled to effectively capture the contextual information in long-term videos, and by integrating temporal attention and spatial attention, the temporal dynamic information of the video is retained, further enhancing the attention to salient areas; by introducing feature mapping and jump connection operations, it can flexibly extract key features from the input data, while enhancing the modeling ability of spatiotemporal features, and making the model adaptable to video inputs of different lengths, ensuring the robustness and expressiveness of the model.

[0037] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0039] Figure 1 2 is a flowchart of the OCVF video understanding model in an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the OCVF video understanding model in an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of the channel attention process in the OCVF model proposed in an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of the process of feature mapping and skip connection in the OCVF model proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that the following detailed descriptions are exemplary only and are intended to describe specific embodiments and provide further explanation of the present invention, and are not intended to limit the exemplary embodiments according to the present invention. Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0044] Example 1

[0045] To address the problem of how to extract important features from videos to generate high-quality summaries, this embodiment proposes a video summary generation method based on feature expression enhancement. By integrating advanced temporal modeling and attention strategies, a new video understanding model based on the improved GoogleNet architecture, the OCVF model, is constructed. The model introduces a channel attention mechanism, which adaptively adjusts the weights of feature channels to effectively highlight key information highly relevant to the video content while suppressing the interference of redundant or irrelevant information, thereby improving the effect of video understanding. Feature mapping operations are introduced to flexibly adjust the feature distribution in the time and space dimensions to more accurately capture dynamic changes and strengthen the fusion of global and local information, thereby enhancing the modeling ability of spatiotemporal features. Feature mapping also enables the model to adapt to video inputs of different lengths, ensuring the robustness and expressiveness of the model. The model is comprehensively designed in terms of feature optimization and information screening to improve its adaptability to tasks and ensure that the generated video summaries are more accurate and richer in information.

[0046] This embodiment proposes a method for generating video summaries based on feature expression enhancement, such as Figure 1 As shown, including:

[0047] Step S1: Obtain the original video and extract the video frame image sequence.

[0048] Step S2: Input the video frame image sequence into the OCVF video understanding model. By utilizing the constructed OCVF video understanding model, the important semantics and key features in the video are effectively extracted to generate the final video summary result to improve the quality of video summary generation.

[0049] Specifically, in the OCVF video understanding model, step S2.1 first receives the input video frame image, and each input video frame image is first subjected to a feature extraction module to extract low-level visual features, i.e., initial features (the features refer to feature maps). In this embodiment, the feature extraction module adopts a convolutional neural network (CNN).

[0050] Step S2.2: The initial features are processed by the attention module to extract key spatiotemporal features. The attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer, and a spatiotemporal attention layer. Figure 2 As shown, where:

[0051] In step S2.2.1, the initial features are adaptively weighted by the channel attention layer to generate key features. By introducing this channel attention mechanism, the weights of feature channels are adaptively adjusted, strengthening key features related to the target task while suppressing unimportant information channels, thereby enhancing the ability to extract key video information.

[0052] Step S2.2.2: The key features are processed by the average pooling layer for dimensionality reduction. By reducing the dimensionality of the extracted features, the spatial dimension is compressed to provide a more compact feature representation.

[0053] Step S2.2.3: The temporal and spatial position information of the video frames is embedded into low-dimensional features through the position encoding layer to help the model better understand the temporal relationship and structural characteristics between video frames.

[0054] In step S2.2.4, the low-dimensional features embedded with position information are processed through the self-attention layer and the spatiotemporal attention layer to extract key spatiotemporal features. The self-attention mechanism captures important global contextual information in long-time videos, further enhancing understanding of them. Simultaneously, the spatiotemporal attention mechanism jointly models the temporal and spatial dimensions, extracting key spatiotemporal features from the video. This effectively models the global spatiotemporal relationships of the video.

[0055] Step S2.3: The extracted multiple features (such as initial features, multi-level intermediate features extracted by the attention mechanism, key spatiotemporal features, etc.) are gradually fused through value embedding feature mapping and jump connection, and the fused features are nonlinearly transformed and optimized to generate a more representative high-level feature representation.

[0056] In the above process, the various features extracted include initial features x , key spatiotemporal features key and the multi-level intermediate features extracted from the attention module (such as key features x_out , features after self-attention mechanism x_att etc.), through jump connection operations such as KC (key connection), CF (current feature connection), IF (intermediate feature connection), etc., multiple features are gradually integrated to enhance the expression ability of spatiotemporal features. Moreover, this process not only integrates the initial features of the original input with the features weighted by the attention mechanism, but also optimizes the features through subsequent multi-layer processing (such as multi-layer perceptron MLP), thereby generating richer and more semantically expressive high-level feature representations.

[0057] Step S2.4: The generated high-level feature representation is then input to the hybrid module, which is actually a classifier. In this embodiment, a multi-layer perceptron (MLP) is used, which includes a fully connected layer, a normalization layer, SigmoidActivation function layer, classification layer, high-level feature representation first passes through the fully connected layer to perform feature mapping to learn frame-level importance information; then the mapped features are standardized by the normalization layer to stabilize the feature distribution and enhance the robustness of the model; finally, Sigmoid The activation function converts the normalized high-level features into a continuous importance score between 0 and 1, which reflects the importance of each video frame. The importance score of each video frame is output, and the key frames are filtered out according to the score to finally generate the video summary result.

[0058] The entire process of the above-mentioned OCVF video understanding model effectively improves the model's ability to model the global key features of the video through mechanisms such as channel attention, self-attention, and spatiotemporal attention. At the same time, combined with position encoding and feature mapping, it effectively enhances the accuracy and information expression ability of video summary generation, ultimately achieving the generation of higher-quality video summaries.

[0059] The following content provides a more detailed introduction to each module in the OCVF video understanding model.

[0060] (1) Channel Attention Mechanism

[0061] The OCVF model proposed in this embodiment is extended based on the existing GoogleNet framework, improves its Inception module, and introduces a new mechanism to adjust the response of different channels. Specifically, the channel attention mechanism adopted by the channel attention layer is as follows: Figure 3 As shown:

[0062] First, the initial feature map is input, and the spatial information of the initial feature map is compressed through global average pooling to obtain a global description of each channel.

[0063] Specifically, after the global average pooling operation, we get c Global description of channels ,for:

[0064] ;

[0065] Among them, the initial feature map of the input X∈R C×H×W , C Indicates the number of channels, H and W Represent the height and width of the feature map respectively, Indicates the first c Channels at position The value of .

[0066] Secondly, 1×1 convolution is used to learn channel attention information, and the activation function is used to obtain the channel attention weight (or weight matrix) of each channel.

[0067] Specifically, the global description is passed through two fully connected layers, using and As an activation function to generate the importance weight of each channel ,for:

[0068] ;

[0069] in, and is a learnable weight matrix, through and Activation function obtains channel importance weight .

[0070] Finally, the channel importance weights are combined with the initial feature map of the original input to obtain the feature map of the specific channel attention, that is, the key feature map, thereby enhancing the effectiveness of feature expression.

[0071] Specifically, by transforming the initial feature map of the original input X Multiply the generated channel weight element by element to obtain the channel attention weighted feature map ,for:

[0072] ;

[0073] Through the above-mentioned channel attention mechanism, the features of each channel will be enhanced or suppressed according to their importance, thereby highlighting the key features.

[0074] (2) Average pooling

[0075] The average pooling layer uses the maximum pooling strategy to reduce the computational burden of subsequent layers by reducing the feature dimension while retaining the most significant feature information. The maximum pooling operation is defined as:

[0076] ;

[0077] in, Indicates that the pooling operation is performed in Layer Position the result; Window Represents the area covered by the pooling operation; Represented in the feature map The index position after pooling operation is translated to a new location ( ) feature map, that is is the index in the current feature map, is the offset in the pooling operation window; max Indicates taking the maximum value.

[0078] In the OCVF model, the network is constructed gradually from shallow to deep layers, initially extracting simple features such as edges and textures. Building deeper features enables the identification of more complex objects and scene structures. This layered approach enables the network to effectively process and analyze complex information in video data. Therefore, through the aforementioned continuous convolution and pooling, the OCVF model is able to construct a hierarchical representation from low-level to high-level features, providing rich and useful features for the subsequent attention mechanism. These features are the foundation for efficient video content analysis and understanding, providing support for the model in video summarization tasks.

[0079] (3) Positional encoding

[0080] In the OCVF model, position encoding aims to provide the model with temporal and spatial position information, so as to better capture the temporal relationship and spatial features between video frames. Specifically, for each input video frame image, the video feature representation ,in, T Indicates the number of frames, Represents the feature dimension of each frame. The position encoding is the feature of each video frame. After the position information of the video frame in the original video is introduced, the feature representation after position encoding is obtained. , and its calculation formula is:

[0081] ;

[0082] in, It is a position encoding matrix, which is used to represent the position information. The position encoding matrix can be calculated using different encoding methods such as fixed positional encoding (FPE), relative positional encoding (RPE), and learned positional encoding to give clear position information to the features of each frame.

[0083] (3.1) Fixed position encoding (FPE) is: Position encoding The calculation is done using the predefined sine and cosine functions, which are of the form:

[0084] ;

[0085] in, t is the time step, i is the index of the feature dimension, Represents the feature dimension of each frame.

[0086] (3.2) Relative Position Encoding (RPE) is: Relative position encoding is calculated based on the relative distance between frames. Let the relative distance be , the corresponding encoding form is:

[0087] ;

[0088] in, is a mapping function, which can be a linear function or other design function.

[0089] (3.3) Learning Position Encoding (LPE) is to enable the model to adaptively learn the position information of each frame through a parameterized position encoding vector, which is in the form of:

[0090] ;

[0091] in, represents the position vector learned by the model.

[0092] Fixed position encoding provides a fixed encoding structure, suitable for positional relationships that do not change during training. Relative position encoding focuses more on the relative relationships between frames, suitable for capturing the temporal dynamics of video frames. Learning position encoding adaptively learns position information for each frame through a model, suitable for more flexible scenarios. Depending on the specific use case, different position encoding methods can be selected to achieve optimal results. In this embodiment, fixed position encoding (FPE) is used.

[0093] After the feature is combined with the position encoding, the position encoding feature The resulting image is fed into a subsequent module for processing. This module not only helps the model better understand the temporal relationship between frame sequences, but also enhances the modeling capability of global context, providing important support for video summary generation.

[0094] (4) Self-attention mechanism

[0095] The self-attention mechanism is used to model the global dependencies between video frames and effectively capture the contextual information in long time series videos. , the attention weight matrix is ​​calculated by the self-attention mechanism, that is, the self-attention mechanism first maps it to the query ( ),key( ) and value( The matrix is:

[0096] ;

[0097] in, To learn the parameters.

[0098] Then, the attention weight is obtained by calculating the correlation between the query and the key:

[0099] ;

[0100] in, is the attention weight matrix, softmax The operation ensures that the weight of each row sums to 1. It is a normalization factor used to alleviate the problem of excessively large dot product values.

[0101] Then, using the attention weight matrix Pair value vector Perform weighted summation to obtain the updated feature representation, which is:

[0102] ;

[0103] The updated features obtained by the above method By integrating global contextual information and encoding important relationships between time steps into features, the model's ability to understand the global characteristics of the video is enhanced. Through the self-attention mechanism, the model can focus on key relationships between frames in the video, effectively enhancing its ability to model global context and providing important support for video summarization.

[0104] (5) Spatiotemporal Attention Mechanism

[0105] In order to integrate the information of temporal attention and spatial attention, this embodiment adopts a weighted fusion strategy to combine the two attention distributions to generate the final attention feature. This fusion strategy not only retains the temporal dynamic information of the video, but also enhances the attention to the salient areas.

[0106] Specifically, the updated feature representation is input as the input feature to the spatiotemporal attention layer. The temporal and spatial attention weight matrices are first calculated through the temporal and spatial attention mechanisms, respectively, and are expressed as and Based on the two attention weight matrices, a dynamic weight adjustment mechanism is used to perform feature weighted fusion to obtain key spatiotemporal features. The fused feature representation can be calculated using the following formula:

[0107] ;

[0108] in, and It is a hyperparameter, and the ratio of the two can be adjusted according to the specific task scenario; Represents the input features.

[0109] In order to further improve the flexibility and adaptability of the attention mechanism, this embodiment also adopts a dynamic weight adjustment mechanism, which is:

[0110] First, the scaling factor ( Scaling Factor ), the calculation formula is:

[0111] ;

[0112] Secondly, according to the scaling factor, the weight of the feature weighted fusion is adaptively adjusted. Specifically, according to the scaling factor calculated in the previous step (abbreviated as SF ), the factor SF is used to adjust the original time weight and spatial weights ,like or , so that the relative importance between the two can be adaptively adjusted according to the spatiotemporal structure of the video, thereby balancing the importance of time and space.

[0113] Through the above dynamic adjustment mechanism, when there are significant differences in the number of time frames (T) and spatial dimensions (D) of the video, the weights will be adaptively adjusted to balance the importance of the two. This dynamic adjustment mechanism is particularly prominent when the video length is long or the feature dimension is large.

[0114] Through the above process, key spatiotemporal features containing key video information can be extracted from the original video frames. This feature is extracted and enhanced through multiple attention mechanisms such as channel attention, self-attention, and spatiotemporal attention. It can effectively capture the interaction of temporal dynamics and spatial information, highlight the key information in the video, and enhance the feature expression ability.

[0115] (6) Feature Mapping and Skip Connections

[0116] In the OCVF model, Value Embedding is used as a feature mapping method to express and optimize features. Specifically, the fully connected layer (which can be regarded as part of the MLP) is used to embed the input features. x Nonlinear mapping can generate more discriminative and complex feature representations. The process can be described as:

[0117] ;

[0118] in, ReLU is the activation function, and are the weights and biases of the MLP.

[0119] In this embodiment, feature mapping is performed in combination with the above value embedding operation. Figure 4 As shown in Figure 1, first, the extracted key spatiotemporal features (key) are fused with features at different levels through different types of jump connections (such as KC, CF, IF), so that features from different sources complement each other and enhance the understanding of video content. Among them, KC (Key Connection) refers to directly fusing features x_att and key Add to enhance the representation ability of features; CF (ConnectionFusion, current feature connection) refers to the feature x_out and x_att Add, so that the final feature contains more semantic information; IF (Input Fusion, intermediate feature connection) refers to the feature x_out With the initial features x Fusion is performed to retain the original feature information and enhance the robustness of the model. Through the above jump connection operation, the final feature y , for use in subsequent video summary generation tasks.

[0120] Secondly, the features obtained by skip connection y and the features obtained by nonlinear mapping Fusion, the fusion feature Z is obtained, which can be expressed as:

[0121] y ;

[0122] Among them, ⊕ represents the feature fusion operation.

[0123] The final fused features generated through the aforementioned feature mapping and skip connections enhance the model's feature expression capabilities, improve information fluidity, prevent gradient vanishing, and promote information fusion through connections between different feature layers. Combined with these feature mapping and fusion, the model can more effectively capture complex relationships in the input data, thereby enhancing overall feature expression capabilities and modeling performance.

[0124] Furthermore, the generated fusion feature Z is subjected to multiple nonlinear mapping and optimizations to generate a more representative, recognizable, and discriminative embedding representation, thereby further improving the modeling ability of the video temporal and spatial information, and ultimately providing stronger feature support for the subsequent video summary generation task. The process is as follows:

[0125] ;

[0126] in, and are weights and biases.

[0127] The aforementioned mapping operation improves the model's ability to capture relationships between different features and express video content. The OCVF model can effectively characterize key features in the input data and achieve significant performance improvements in comprehensive modeling. Furthermore, this judgment and mapping operation is performed directly in the forward propagation, avoiding redundant auxiliary function calls, making the entire process more efficient and clear, and providing a good foundation for subsequent expansion to deeper nonlinear mappings. Furthermore, the integration with other modules further enhances the model's ability to model spatiotemporal features. In other words, feature mapping plays a core role in the OCVF model, enabling it to better understand and express video content.

[0128] (7) Classifier

[0129] In the OCVF model, the classifier optimizes the importance scoring process of video frames through a carefully designed layer structure and activation function. The classifier consists of two key parts: linear transformation and nonlinear activation, and layer normalization is used to enhance the stability and efficiency of the model.

[0130] In the classifier, the input features are first projected into an intermediate feature space through a fully connected linear layer to capture key information. This process can be expressed as:

[0131] ;

[0132] in, and are the weights and biases of this layer.

[0133] Then, the ReLU activation function is used to introduce nonlinearity, which helps to enhance the model's ability to learn complex relationships and can be expressed as:

[0134] ;

[0135] The above ReLU activation function helps to prevent the activation value from being too large while keeping the gradient flowing, thereby maintaining the stability of the network.

[0136] Furthermore, in order to control overfitting and improve the generalization ability of the model, the Dropout layer is applied to randomly discard some activation values, which can be expressed as:

[0137] ;

[0138] in, Indicates that there is a 50% probability of discarding the output of a neuron.

[0139] After that, the second fully connected layer converts the processed features into the final classification score, which is mapped to the [0, 1] interval through the Sigmoid function to indicate the importance of each video frame, which can be expressed as:

[0140] ;

[0141] Where σ represents Sigmoid function, and are the weights and biases of the layer.

[0142] Finally, the classifier outputs the predicted frame importance score ,Should is used to calculate the loss function, and the true label is In this embodiment, the loss function includes two parts: classification loss and regression loss. The classification loss is expressed using cross entropy loss as follows:

[0143] ;

[0144] The regression loss is expressed using the mean squared error (MSE) as:

[0145] ;

[0146] The total loss is weighted combination:

[0147] ;

[0148] in, and is a weight parameter that balances classification and regression losses. This loss function optimizes the classifier's ability to score frame importance while improving the overall performance of the model. N Indicates the frame number.

[0149] In addition, in order to ensure the numerical stability of the output of each layer of the network, the layer normalization technology is used to standardize the output of each neuron, which can be expressed as:

[0150] ;

[0151] Among them, μ and σ represent the mean and standard deviation of the output of the current layer, respectively, and and are learnable scaling and translation parameters.

[0152] Through the above design, the classifier can not only effectively evaluate the importance of each frame of video, but also ensure the effective transmission and integration of features in the network through the integration of layer normalization and skip connections, accelerate the convergence of the model, and improve the stability of training.

[0153] Furthermore, the correlation coefficient is used: Kendall's Tau ( τ ) and Spearman's Rank Correlation Coefficient ( ρ ) to evaluate the model, measuring the similarity between the model-generated summaries and the human-annotated summaries. These two correlation coefficients are used to assess the ranking consistency between the generated summaries and the user annotations, thereby verifying the model's effectiveness and reliability in capturing important frames in the video. By calculating these correlation coefficients, we can fully understand the model's ranking consistency performance and ensure the scientific and verifiable quality of the summaries.

[0154] The following experiments are conducted to further verify the effectiveness of the method proposed in this embodiment. Specifically, the experiment is conducted on a server equipped with an NVIDIA RTX 3090 GPU. The model training uses the following parameter settings: epochs (rounds) is 100, batch_size (data size) is set to 1, learning_rate (learning rate) is 1e-3, and weight_decay (weight decay) is 1e-7. These parameters are selected to ensure that the model can converge efficiently in the video summary generation task and avoid overfitting while maintaining model performance. The experimental framework is implemented using PyTorch, combined with GPU acceleration to ensure the efficiency and stability of the training process.

[0155] In the experiments, two standard video summarization datasets, SumMe and TVSum, were used to validate the performance of the OCVF model. The TVSum dataset, designed specifically for video summarization research, contains 50 videos selected from YouTube. These videos cover 10 different categories, including news reports, documentaries, and educational videos. The average video length is approximately 5 minutes, and each video has an average of 20 independent annotations, totaling 1,000 annotations. These annotations score the importance of each shot in the video, reflecting the importance of the video segment to the overall content. The SumMe dataset contains 25 videos covering a variety of topics, including holiday travel, sports activities, and personal activities. The video lengths range from 1 minute to 6 minutes. SumMe uses manual annotation methods, but it provides video summaries rather than importance scores for individual frames or segments. Each video has at least 15 different manually generated summaries, totaling 390 summary annotations. These summaries were created by different annotators and reflect the content considered most important and representative in the video. While SumMe’s data provides valuable insights into the level of detail of manual summaries, it is limited by the small number of videos and annotations, which may affect its applicability in a wider range of scenarios and the effectiveness of algorithm training.

[0156] These two datasets are widely used for video summarization tasks and serve as standard test sets in the current field. Training and evaluation on these two datasets allows us to fully verify the performance and adaptability of the OCVF model across different video types.

[0157] In order to evaluate the performance of the OCVF model in the video summary generation task, we selected several existing models such as HSA-RNN, VJMHT, RR-STG, MSVA, SSPVS, GoogleNet, CSTA and DMASum (the above network models are all existing models and will not be repeated here) for specific comparison. These models cover a range from traditional feature extraction methods to the latest technologies based on deep learning, representing the development direction of the video summarization field.

[0158] Compared to HSA-RNN, OCVF has superior feature modeling capabilities. By combining multimodal features with recurrent neural networks, HSA-RNN is able to capture the temporal characteristics of videos to a certain extent. However, recurrent neural networks are susceptible to the vanishing gradient problem when processing long temporal relationships and their capture of dynamic features between frames is relatively crude. OCVF, through improved feature mapping operations, not only enhances its ability to model temporal relationships between video frames but also improves the expression of correlations between multimodal features, resulting in superior performance in video summarization tasks.

[0159] VJMHT, based on a hidden Markov tree modeling approach, is suitable for video summarization of structured scenes. Its advantage lies in its ability to model the hierarchical nature of videos through a tree structure. However, this approach is limited in its ability to capture global contextual relationships in complex dynamic scenes. OCVF, by comprehensively modeling spatiotemporal features, can accurately capture key frames in complex dynamic videos and effectively integrate key information from different modalities within the video, thus outperforming VJMHT in complex scenarios.

[0160] RR-STG utilizes spatiotemporal graph modeling to explicitly describe the temporal dynamics of videos. This approach has a strong advantage in preserving temporal relationships in videos. However, the high-dimensional nature of spatiotemporal graphs limits its performance when processing large-scale video data. OCVF effectively addresses efficiency issues in feature extraction and temporal modeling by introducing a channel-wise attention mechanism and a flexible skip connection strategy. This approach achieves higher computational efficiency and scalability while maintaining performance.

[0161] MSVA uses a multimodal attention mechanism to enhance feature extraction capabilities of video content by focusing on the connections between multiple modalities. However, it has weak feature representation for non-salient frames and tends to overlook key dynamic changes in complex scenes. OCVF uses a channel-wise attention mechanism to adaptively adjust the importance of different channels, enabling the model to highlight important dynamic features in the video, and excels in diverse video scenarios.

[0162] SSPVS is a video summarization method that utilizes saliency and structural features. Although it performs well in static scenes, its effectiveness is limited when dealing with dynamic and complex scenes. In contrast, OCVF achieves more refined feature capture in dynamic scenes by optimizing feature mapping operations and attention mechanisms, significantly improving the quality of video summarization.

[0163] GoogleNet, a classic convolutional neural network, possesses strong feature extraction capabilities, but its design is not optimized for the spatiotemporal characteristics of video data. OCVF improves on the GoogleNet architecture by introducing a channel attention module after each Inception module, significantly enhancing its ability to model spatiotemporal features of videos, thereby achieving superior performance in video summarization tasks.

[0164] CSTA improves its understanding of video content by incorporating a spatiotemporal attention mechanism, but its feature representation is less granular, particularly in modeling cross-modal feature interactions. OCVF, building on the spatiotemporal modeling strengths of the CSTA framework, further enhances its performance in capturing multimodal feature correlations through channel attention and numerical embedding modules.

[0165] DMASum, based on the extraction and aggregation of deep features, performs well in specific scenarios, but its generalization ability in cross-domain video tasks is limited. OCVF emphasizes in-depth modeling of visual features in its design and enhances its understanding of temporal and spatial features through feature mapping operations, thus showing greater robustness and adaptability in summarization of diverse video types.

[0166] To verify the effectiveness of OCVF in video summarization, we compared it with existing methods, such as HSA-RNN, VJMHT, RR-STG, MSVA, SSPVS, GoogleNet, CSTA, and DMASum, before channel attention was applied to video summarization models. The experimental results are shown in Table 1. These results demonstrate that OCTV demonstrates significant advantages on the SumMe dataset, achieving state-of-the-art performance, and maintains a high score on the TVSum dataset, demonstrating its stability and generalization capabilities.

[0167] Table 1: Comparison of results of different models

[0168]

[0169] Among them, τ and ρ are two correlation coefficients for measuring ranking consistency. τ is used to compare the pairwise consistency between two ranking sequences and is suitable for measuring the stability of rankings; ρ is calculated based on the rank values ​​of the two variables, reflecting the overall consistency between the rankings and is suitable for evaluating the strength of the correlation between variables.

[0170] In contrast, while DMASum achieved strong performance on the TVSum dataset, its score on the SumMe dataset was significantly lower, demonstrating its limited adaptability to diverse datasets. OCVF, by integrating a channel-wise attention mechanism and feature mapping operations, demonstrates a more comprehensive and efficient approach to video feature extraction and temporal modeling, capable of extracting keyframes in both dynamic and complex scenes. In particular, OCVF's balanced performance across both datasets demonstrates its broad applicability across diverse video types and content.

[0171] Furthermore, OCVF's design emphasizes in-depth modeling of visual dependencies. Compared to traditional methods that focus solely on temporal dependencies, OCVF incorporates a channel-wise attention mechanism within each feature module. This not only captures critical visual context but also enhances scene understanding through the absolute position of frames. This capability enables OCVF to excel in capturing key moments and generating highly generalizable summaries. In particular, OCVF's performance on the SumMe dataset further demonstrates its robustness in complex and diverse videos.

[0172] In addition, OCVF also has the advantages of high efficiency and simplicity. Unlike methods that require additional modal data or specific task features, OCVF learns the feature relationship between frames through a deep convolutional network, combined with feature mapping operations and attention strategies, to achieve higher performance at a lower computational cost. This design enables OCVF to outperform existing methods in terms of τ and ρ indicators, and demonstrate its unique advantages and innovations in the task of video summary generation.

[0173] In addition, this experiment also designed an ablation experiment to verify the contribution of the main modules of the model (method) to the overall performance. The channel attention mechanism and feature mapping operation (feature mapping operation) were removed respectively. The feature mapping operation specifically refers to the value embedding operation. The performance on the SumMe and TVSum datasets was compared. The experimental results are shown in Table 2 below.

[0174] Table 2: Ablation experiment results

[0175]

[0176] Specifically, after removing the channel attention mechanism, the model's τ and ρ values ​​on the SumMe dataset dropped from 0.261 and 0.290 to 0.249 and 0.278, respectively. On the TVSum dataset, τ and ρ values ​​dropped from 0.198 and 0.260 to 0.191 and 0.251, respectively. This demonstrates that the channel attention mechanism plays a crucial role in cross-channel feature fusion, better capturing the complex correlation information in video data.

[0177] Furthermore, the improved feature mapping operation played an important role in the model. On the SumMe dataset, the τ and ρ values ​​were 0.246 and 0.275, respectively; on the TVSum dataset, the τ and ρ values ​​were 0.195 and 0.256, respectively. This indicates that the feature mapping operation has a positive impact on model performance by enhancing feature expression capabilities, effectively enhancing feature expression capabilities.

[0178] The above ablation experiments demonstrate that feature mapping plays a significant role in improving feature representation, while other modules provide additional support for optimizing model performance. Feature mapping effectively transforms input data through multi-layer nonlinear mapping and feature fusion operations, thereby enhancing the model's ability to represent key features. Compared to traditional feature processing methods, the improved feature mapping operation places greater emphasis on fine-grained feature representation, enabling more accurate capture of spatiotemporal information changes and key dynamic features in videos. Experimental results demonstrate that OCVF, incorporating the improved feature mapping operation, achieves significant performance improvements on both the SumMe and TVSum datasets. When identifying dynamic scenes and key objects, the improved feature mapping enables the model to better focus on important features, thereby improving the accuracy and overall quality of video summarization. Compared to traditional models, it demonstrates greater robustness and detail capture, particularly in diverse and complex scenes.

[0179] The channel attention mechanism plays a core role in capturing the correlation of multi-channel features, while feature mapping operations provide additional support for feature representation. The channel attention module calculates the features of each channel through global average pooling and generates channel weights through two fully connected layers (ReLU and Sigmoid activation functions), thereby achieving channel-by-channel weight adjustment. This module enhances the model's response to salient features, suppresses unimportant features, and significantly improves the effectiveness of feature representation. Compared with spatial attention or global attention, channel attention focuses more on fine-grained channel-level feature selection and can better model the dynamic changes in spatiotemporal information in videos. Experiments show that OCVF with the addition of the channel attention module achieves significant performance improvements on the SumMe and TVSum datasets. When identifying dynamic scenes and important objects, the channel attention mechanism enables the model to focus more on significantly changing features, thereby improving the accuracy and quality of video summary generation. Compared with traditional models, OCVF demonstrates stronger performance in Kendall's Tau (τ) and Spearman's Rank Correlation (ρ) indicators, especially in diverse and complex scenarios, showing stronger generalization and detail capture capabilities. The complete OCVF model achieves optimal performance by integrating these modules.

[0180] In summary, the new OCVF-based video summarization model proposed in this embodiment addresses the problems of existing methods in insufficient key feature extraction in multimodal videos and their understanding of the spatiotemporal relationships between video features. OCVF introduces a channel attention mechanism that can adaptively adjust feature channel weights. Combined with multiple position encoding strategies and feature mapping operations, it significantly improves the model's understanding of video content and its ability to synthesize features. Experimental results on the standard datasets SumMe and TVSum demonstrate that OCVF significantly improves performance compared to current mainstream methods, demonstrating its robustness and wide applicability in complex scenes and diverse video types. This OCVF model provides an effective and efficient solution for video summarization. Its innovative design is not only of great theoretical significance but also provides new directions for the further development of video understanding and generation technologies.

[0181] Example 2

[0182] This embodiment provides a video summary generation system based on feature expression enhancement, including:

[0183] Video acquisition module, used to acquire original video and extract video frame image sequence;

[0184] The video summary generation module is used to input the video frame image sequence into the OCVF video understanding model. Each input video frame image is first extracted by the feature extraction module to extract the initial features. The initial features are then used by the attention module to extract key spatiotemporal features. Then, high-level feature representations are generated through feature mapping and skip connections. Finally, the high-level feature representations output the importance score of each video frame through the classifier, determine the key frames, and generate the final video summary result.

[0185] Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer arranged in sequence; the initial features are adaptively adjusted to the channel weights by the channel attention layer to generate key features; after the dimensionality reduction processing by the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted by the self-attention layer and the spatiotemporal attention layer.

[0186] Example 3

[0187] This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for implementing the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0188] Example 4

[0189] This embodiment further provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above method provided in this embodiment.

[0190] Example 5

[0191] This embodiment provides a computer program product including executable instructions, which are computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method provided in this embodiment.

[0192] The steps involved in the above embodiments 2 to 5 correspond to those in embodiment 1. For detailed implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0193] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0194] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention is described in conjunction with the accompanying drawings, it does not limit the scope of protection of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A video summary generation method based on feature expression enhancement, characterized in that: include: Get the original video and extract the video frame image sequence; The video frame image sequence is input into the OCVF video understanding model. Each input video frame image is first extracted by the feature extraction module to extract the initial features. The initial features are then used by the attention module to extract key spatiotemporal features. Then, high-level feature representations are generated through feature mapping and skip connections. Finally, the high-level feature representations output the importance score of each video frame through the classifier, determine the key frames, and generate the final video summary result. The attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer, and a spatiotemporal attention layer, which are arranged in sequence. The initial features are adaptively adjusted in the channel weights by the channel attention layer to generate key features. After dimensionality reduction processing by the average pooling layer, the temporal and spatial position information of the video frame is embedded into low-dimensional features by the position encoding layer. Finally, the key spatiotemporal features are extracted by the self-attention layer and the spatiotemporal attention layer. In the position coding layer, for the feature representation of each input video frame image, the position information of the current video frame in the original video is introduced to obtain the feature representation after position coding; The position-encoded features are sequentially passed through the self-attention layer and the spatiotemporal attention layer to extract key spatiotemporal features, including: the position-encoded features are input as input features to the self-attention layer, the attention weight matrix is ​​calculated through the self-attention mechanism, and the updated feature representation is obtained by weighted summation of the attention weight matrix; the updated feature representation is then input as input features to the spatiotemporal attention layer, and the temporal and spatial attention weight matrices are calculated through the temporal and spatial attention mechanisms, respectively, which are expressed as and Based on the two attention weight matrices and a dynamic weight adjustment mechanism, feature weighted fusion is performed to obtain key spatiotemporal features. The fused feature representation is calculated using the following formula: ； in, and is a hyperparameter; Represents input features; The dynamic weight adjustment mechanism is: According to the time frame number and spatial dimension of the original video, the scaling factor SF is calculated as follows: According to the calculated scaling factor SF, the scaling factor SF is used to adjust the original time weight and spatial weights ; The feature map and skip connection are: Use multi-layer perceptron to perform nonlinear mapping on input features to generate complex feature representations; Through jump connections, the complex feature representation is fused with the extracted initial features, multi-level intermediate features, and key spatiotemporal features to obtain fused features; among them, the multi-level intermediate features include key features; the fused features are then subjected to multiple nonlinear mappings to generate the final embedded representation, i.e., the high-level feature representation.

2. The method for generating a video summary based on feature expression enhancement according to claim 1, wherein: In the channel attention layer, the initial input features are subjected to global average pooling to compress the spatial information to obtain a global description of each channel, and then the channel attention information is learned through convolution, and the activation function is used to obtain the attention weight of each channel; Based on the attention weight of each channel, the original initial features of the input are multiplied element by element to obtain the channel-weighted features, i.e., the key features.

3. The method for generating a video summary based on feature expression enhancement according to claim 1, wherein: The position information is represented by a position coding matrix, which is calculated using any one or more of fixed position coding, relative position coding, and learned position coding.

4. A video summary generation system based on feature expression enhancement, using the video summary generation method based on feature expression enhancement according to claim 1, characterized in that: include: Video acquisition module, used to acquire original video and extract video frame image sequence; The video summary generation module is used to input the video frame image sequence into the OCVF video understanding model. Each input video frame image is first extracted by the feature extraction module to extract the initial features. The initial features are then used by the attention module to extract key spatiotemporal features. Then, high-level feature representations are generated through feature mapping and skip connections. Finally, the high-level feature representations output the importance score of each video frame through the classifier, determine the key frames, and generate the final video summary result. Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer arranged in sequence; the initial features are adaptively adjusted to the channel weights by the channel attention layer to generate key features; after the dimensionality reduction processing by the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted by the self-attention layer and the spatiotemporal attention layer.

5. An electronic device, characterized in that: include: a memory for storing executable instructions; The processor is configured to implement the method for generating a video summary based on feature expression enhancement according to any one of claims 1 to 3 when executing the executable instructions stored in the memory.

6. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause a processor to execute the executable instructions to implement the video summary generation method based on feature expression enhancement according to any one of claims 1 to 3.

7. A computer program product, characterized in that The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the method for generating a video summary based on feature expression enhancement according to any one of claims 1 to 3 is implemented.

Images