Video abstract generation method and system based on feature expression enhancement

Abstract

The invention discloses a video abstract generation method and system based on feature expression enhancement, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an original video, and extracting a video frame image sequence; a video frame image sequence is input into an OCVF video understanding model, initial features of each input video frame image are extracted through a feature extraction module, key spatio-temporal features of the initial features are extracted through an attention module, and then high-level feature representation is generated through feature mapping and jump connection. The high-level feature representation outputs the importance score of each video frame through a classifier, key frames are determined, and a final video abstract result is generated; wherein the attention module comprises a channel attention layer, an average pooling layer, a position coding layer, a self-attention layer and a space-time attention layer which are arranged in sequence. According to the method, more comprehensive modeling and efficient understanding of the video content can be realized, and more robust and more accurate video abstract generation is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for generating a video summary based on feature expression enhancement. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Video summarization is an important task in the field of natural language processing (NLP) and is also a current research hotspot. With the rapid development of short video platforms, video data has exploded. How to efficiently understand and extract key information from videos and generate concise and accurate video summaries is one of the core issues that need to be solved. Video summarization refers to extracting key information from videos and compressing long videos into short and easy-to-understand forms so that users can quickly obtain the core content of the video.

[0004] At present, video summarization mainly focuses on extracting key information, especially extracting and summarizing the core content of the video through key frames. However, this method often has many shortcomings. For example, existing research usually analyzes the key features of video frames to compress video information and retain important elements, but the extracted features of video frames are often not representative enough. With the development of neural networks, considering the powerful feature extraction ability of CNN (Convolutional Neural Networks, convolutional neural network), CNN is often used to extract features of video frames for in-depth analysis and understanding, so as to improve the representativeness of video frame features. However, this method does not fully utilize the spatial position information in the video, resulting in incomplete feature extraction and poor accuracy of the generated video summary.

[0005] In addition, it is still difficult to fully capture the complex spatiotemporal relationship by relying solely on convolutional neural networks to extract features from video frames. In order to enhance the modeling ability of spatiotemporal features of videos, the current method is to introduce temporal convolution or attention mechanisms to enhance the modeling ability of spatiotemporal features, and then combine the extracted features to gain a more comprehensive understanding of the video. Although the above improvements have achieved good results in specific scenarios, there are still some shortcomings, such as the inability to fully capture the extraction of key features in long-time videos, and the feature expression of different modalities is not fine enough. Summary of the invention

[0006] To address the deficiencies of the above-mentioned prior art, the present invention provides a method and system for generating video summaries based on enhanced feature expression, integrates advanced temporal modeling and attention strategy, and constructs a new video understanding model OCVF (Optimized Channel Value Framework). The OCVF model is used to achieve more comprehensive modeling and efficient understanding of video content, thereby achieving more robust and accurate video summary generation.

[0007] In a first aspect, the present invention provides a method for generating a video summary based on feature expression enhancement.

[0008] A video summary generation method based on feature expression enhancement, comprising: Get the original video and extract the video frame image sequence; The video frame image sequence is input into the OCVF video understanding model. Each input video frame image is first extracted with initial features by the feature extraction module. The initial features are then extracted with key spatiotemporal features by the attention module. Then, high-level feature representation is generated through feature mapping and skip connection. Finally, the high-level feature representation outputs the importance score of each video frame through the classifier, determines the key frames, and generates the final video summary result. Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer which are arranged in sequence; the initial features are adaptively adjusted through the channel weights of the channel attention layer to generate key features; after the dimensionality reduction processing through the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted through the self-attention layer and the spatiotemporal attention layer.

[0009] In a further technical solution, in the channel attention layer, the input initial feature map is subjected to global average pooling to compress the spatial information to obtain a global description of each channel, and then the channel attention information is learned through convolution, and the attention weight of each channel is obtained using an activation function; Based on the attention weight of each channel, the original initial feature map of the input is multiplied element by element to obtain the channel-weighted feature map, i.e., the key feature map.

[0010] A further technical solution is that in the position coding layer, for the feature representation of each input video frame image, the position information of the current video frame in the original video is introduced to obtain the feature representation after position coding; The position information is represented by a position coding matrix, which is calculated by using any one or more methods of fixed position coding, relative position coding, and learning position coding.

[0011] A further technical solution is to extract key spatiotemporal features from the position-encoded features through the self-attention layer and the spatiotemporal attention layer in turn, including: The position-encoded features are input as input features to the self-attention layer, and the attention weight matrix is ​​calculated through the self-attention mechanism. The updated feature representation is obtained by weighted summation of the attention weight matrix. The updated feature representation is then input as the input feature into the spatiotemporal attention layer. The temporal and spatial attention weight matrices are calculated through the temporal and spatial attention mechanisms respectively. Based on the two attention weight matrices and using a dynamic weight adjustment mechanism, feature weighted fusion is performed to obtain the key spatiotemporal features.

[0012] A further technical solution is that the dynamic weight adjustment mechanism is: The scaling factor is calculated based on the number of time frames and spatial dimensions of the original video; According to the scaling factor, the weights of feature weighted fusion are adaptively adjusted to balance the importance of time and space.

[0013] In a further technical solution, the feature map and the jump connection are: Use multi-layer perceptron to perform nonlinear mapping on input features and generate complex feature representations; Through skip connections, the complex feature representation is fused with the extracted initial features, multi-level intermediate features, and key spatiotemporal features to obtain fused features; wherein the multi-level intermediate features include key features; The fused features are then subjected to multiple nonlinear mappings to generate the final embedded representation, i.e., the high-level feature representation.

[0014] In a second aspect, the present invention provides a video summary generation system based on feature expression enhancement.

[0015] A video summary generation system based on feature expression enhancement, comprising: A video acquisition module is used to acquire the original video and extract the video frame image sequence; The video summary generation module is used to input the video frame image sequence into the OCVF video understanding model. Each input video frame image is first extracted with initial features by the feature extraction module, and the initial features are extracted with key spatiotemporal features by the attention module. Then, high-level feature representation is generated through feature mapping and jump connection. Finally, the high-level feature representation outputs the importance score of each video frame through the classifier, determines the key frame, and generates the final video summary result. Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer which are arranged in sequence; the initial features are adaptively adjusted through the channel weights of the channel attention layer to generate key features; after the dimensionality reduction processing through the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted through the self-attention layer and the spatiotemporal attention layer.

[0016] In a third aspect, the present invention further provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned method for generating a video summary based on feature expression enhancement when executing the executable instructions stored in the memory.

[0017] In a fourth aspect, the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned method for generating a video summary based on feature expression enhancement.

[0018] In a fifth aspect, the present invention also provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned method for generating a video summary based on feature expression enhancement is implemented.

[0019] One or more of the above technical solutions have the following beneficial effects: 1. The present invention provides a method and system for generating video summaries based on enhanced feature expression, integrates advanced temporal modeling and attention strategies, constructs an OCVF video understanding model, and utilizes the OCVF model to achieve more comprehensive modeling and efficient understanding of video content, thereby achieving more robust and accurate video summary generation.

[0020] 2. In the OCVF model proposed in the present invention, by introducing the channel attention mechanism, the importance of each feature channel can be adaptively adjusted to highlight the key features, thereby effectively concentrating the features that are helpful for the task, improving the effect of video understanding, and enhancing the module's ability to express video features; through time series modeling, the model can better understand the temporal relationship of the frame sequence and enhance the modeling ability of the global context; then the global dependency between video frames is modeled through the self-attention mechanism, effectively capturing the contextual information in long time series videos, and by integrating temporal attention and spatial attention, the temporal dynamic information of the video is retained, and the attention to salient areas is further enhanced; by introducing feature mapping and jump connection operations, it can flexibly extract key features from the input data, while enhancing the modeling ability of spatiotemporal features, and enabling the model to adapt to video inputs of different lengths, ensuring the robustness and expressiveness of the model.

[0021] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 It is a workflow diagram of the OCVF video understanding model in an embodiment of the present invention; Figure 2 Schematic diagram of the OCVF video understanding model in an embodiment of the present invention; Figure 3 A schematic diagram of the process of channel attention in the OCVF model proposed in an embodiment of the present invention; Figure 4 Schematic diagram of the process of feature mapping and skip connection in the OCVF model proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0025] Embodiment 1 In order to solve the problem of how to extract important features from a video to generate a high-quality summary, this embodiment proposes a video summary generation method based on feature expression enhancement. By integrating advanced temporal modeling and attention strategies, a new video understanding model based on the improved GoogleNet architecture, the OCVF model, is constructed. The channel attention mechanism is introduced into the model. By adaptively adjusting the weights of feature channels, the key information highly related to the video content is effectively highlighted, while suppressing the interference of redundant or irrelevant information, thereby improving the effect of video understanding; the feature mapping operation is introduced to flexibly adjust the feature distribution in the time and space dimensions, so as to more accurately capture dynamic changes and strengthen the fusion of global and local information, thereby enhancing the modeling ability of spatiotemporal features. Through feature mapping, the model can be adapted to video inputs of different lengths to ensure the robustness and expression ability of the model; the model is comprehensively designed in terms of feature optimization and information screening to improve the adaptability of the model to the task, ensuring that the generated video summary is more accurate and richer in information.

[0026] The video summary generation method based on feature expression enhancement proposed in this embodiment is as follows: Figure 1 As shown, including: Step S1, obtaining the original video and extracting the video frame image sequence.

[0027] Step S2: Input the video frame image sequence into the OCVF video understanding model. By utilizing the constructed OCVF video understanding model, the important semantics and key features in the video are effectively extracted to generate the final video summary result, so as to improve the quality of video summary generation.

[0028] Specifically, in the OCVF video understanding model, step S2.1 first receives the input video frame image, and each input video frame image is first extracted by a feature extraction module to extract low-level visual features, i.e., initial features (the features refer to feature maps). In this embodiment, the feature extraction module uses a convolutional neural network CNN.

[0029] Step S2.2: The initial features are processed by an attention module to extract key spatiotemporal features. The attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer, and a spatiotemporal attention layer. Figure 2 As shown, where: Step S2.2.1: The initial features are adaptively adjusted through the channel attention layer to generate key features. By introducing the channel attention mechanism, the weights of feature channels are adaptively adjusted to strengthen key features related to the target task, while suppressing unimportant information channels, thereby enhancing the ability to extract key video information.

[0030] Step S2.2.2: The key features are processed by the average pooling layer for dimensionality reduction. By reducing the dimensionality of the extracted features and compressing the spatial dimensions, a more compact feature representation is provided.

[0031] Step S2.2.3: The temporal and spatial position information of the video frames is embedded into low-dimensional features through the position encoding layer, so as to help the model better understand the temporal relationship and structural characteristics between video frames.

[0032] Step S2.2.4: The low-dimensional features embedded with position information are used to extract key spatiotemporal features through the self-attention layer and the spatiotemporal attention layer. Through the self-attention mechanism, the global important contextual information in the long time sequence video can be captured, and the ability to understand the long time sequence video can be further enhanced. At the same time, through the spatiotemporal attention mechanism, the time dimension and the spatial dimension are jointly modeled to extract the key spatiotemporal features in the video, so as to effectively model the global spatiotemporal relationship of the video.

[0033] Step S2.3: The extracted multiple features (such as initial features, multi-level intermediate features extracted by the attention mechanism, key spatiotemporal features, etc.) are gradually fused through value embedding feature mapping and jump connection, and the fused features are nonlinearly transformed and optimized to generate a more representative high-level feature representation.

[0034] In the above process, the various features extracted include initial features x , key spatiotemporal features key and the multi-level intermediate features extracted from the attention module (such as key features x_out , features after self-attention mechanism x_att Through jump connection operations such as KC (key connection), CF (current feature connection), IF (intermediate feature connection), etc., multiple features are gradually integrated to enhance the expressiveness of spatiotemporal features. This process not only integrates the initial features of the original input with the features weighted by the attention mechanism, but also optimizes the features through subsequent multi-layer processing (such as multi-layer perceptron MLP), thereby generating richer and more semantically expressive high-level feature representations.

[0035] Step S2.4: The generated high-level feature representation is then input into the hybrid module. The hybrid module is actually a classifier. This embodiment uses a multi-layer perceptron (MLP), which includes a fully connected layer, a normalization layer, Sigmoid Activation function layer, classification layer, high-level feature representation first passes through the fully connected layer for feature mapping to learn frame-level importance information; then the mapped features are standardized by the normalization layer to stabilize the feature distribution and enhance the robustness of the model; finally, SigmoidThe activation function converts the normalized high-level features into a continuous importance score between 0 and 1, which reflects the importance of each video frame. The importance score of each video frame is output, and the key frames are filtered out according to the score to finally generate the video summary result.

[0036] The entire process of the above-mentioned OCVF video understanding model effectively improves the model's ability to model the global key features of the video through mechanisms such as channel attention, self-attention, and spatiotemporal attention. At the same time, it combines position encoding and feature mapping to effectively enhance the accuracy and information expression ability of video summary generation, and ultimately achieves the generation of higher quality video summaries.

[0037] The following content provides a more detailed introduction to each module in the OCVF video understanding model.

[0038] (1) Channel Attention Mechanism The OCVF model proposed in this embodiment is extended on the basis of the existing GoogleNet framework, improves its Inception module, and introduces a new mechanism to adjust the response of different channels. Specifically, the channel attention mechanism adopted by the channel attention layer is as follows: Figure 3 As shown: First, the initial feature map is input, and the spatial information of the initial feature map is compressed through global average pooling to obtain a global description of each channel.

[0039] Specifically, after the global average pooling operation, we get c Global description of channels ,for: ; Among them, the initial feature map of the input X∈R C×H×W , C Indicates the number of channels, H and W Represent the height and width of the feature map respectively, Indicates the feature map c Channels at position The value of .

[0040] Secondly, 1×1 convolution is used to learn the channel attention information, and the activation function is used to obtain the channel attention weight (or weight matrix) of each channel.

[0041] Specifically, the global description is passed through two fully connected layers, using and As an activation function to generate the importance weight of each channel ,for: ; in, and is a learnable weight matrix, through and The activation function obtains the channel importance weight .

[0042] Finally, the channel importance weights are combined with the initial feature map of the original input to obtain the feature map of the specific channel attention, that is, the key feature map, thereby enhancing the effectiveness of feature expression.

[0043] Specifically, by transforming the original input initial feature map X Multiply the generated channel weight element by element to get the channel attention weighted feature map ,for: ; Through the above-mentioned channel attention mechanism, the features of each channel will be enhanced or suppressed according to their importance, thereby highlighting the key features.

[0044] (2) Average Pooling The average pooling layer adopts the maximum pooling strategy to reduce the computational burden of subsequent layers by reducing the feature dimension while retaining the most significant feature information. The maximum pooling operation is defined as: ; in, Indicates that the pooling operation is performed in Layer Position Result; Window Represents the area covered by the pooling operation; Represented in the feature map The index position after the pooling operation is translated to a new location ( ), that is, is the index in the current feature map, is the offset in the pooling operation window; max Indicates taking the maximum value.

[0045] In the OCVF model, considering that the network is gradually constructed from shallow to deep layers, simple features such as edges and textures are initially extracted, while building deeper features can identify more complex objects and scene structures. This hierarchical processing method enables the network to effectively process and analyze complex information in video data. Therefore, through the above-mentioned continuous convolution and pooling, the OCVF model can construct a hierarchical representation from low-level to high-level features, providing rich and useful features for the subsequent attention mechanism. These features are the basis for efficient video content analysis and understanding, and provide support for the model when performing video summarization tasks.

[0046] (3) Positional encoding In the OCVF model, position encoding aims to provide the model with temporal and spatial position information, so as to better capture the temporal relationship and spatial features between video frames. Specifically, for each input video frame image, the video feature representation ,in, T Indicates the frame number. Represents the feature dimension of each frame. The position encoding is the feature of each video frame. After introducing the position information of the video frame in the original video, the feature representation after position encoding is obtained , and its calculation formula is: ; in, It is a position encoding matrix, which is used to represent the position information. The position encoding matrix can be calculated using different encoding methods such as fixed positional encoding (FPE), relative positional encoding (RPE), and learned positional encoding to give clear position information to the features of each frame.

[0047] (3.1) Fixed position encoding (FPE) is: Position encoding The calculation is done using the predefined sine and cosine functions, which have the form: ; in, t is the time step, i is the index of the feature dimension, Represents the feature dimension of each frame.

[0048] (3.2) Relative position encoding (RPE) is: The relative position encoding is calculated based on the relative distance between frames. Let the relative distance be , the corresponding encoding form is: ; in, is a mapping function, which can be a linear function or other design functions.

[0049] (3.3) Learning position encoding (LPE) is: through the parameterized position encoding vector, the model can adaptively learn the position information of each frame in the form of: ; in, represents the position vector learned by the model.

[0050] The fixed position encoding is used to provide a fixed encoding structure, which is suitable for positional relationships that do not change with training; relative position encoding focuses more on the relative relationship between frames, which is suitable for capturing the temporal dynamics of video frames; and learning position encoding learns the position information of each frame through model adaptive learning, which is suitable for more flexible scenarios. Depending on the specific usage scenario, different position encoding methods can be selected to achieve better results. In this embodiment, fixed position encoding FPE is selected for application.

[0051] After the feature is combined with the position encoding, the position encoding feature It is sent to the subsequent module for processing. This module not only helps the model better understand the temporal relationship of the frame sequence, but also enhances the modeling ability of the global context, providing important support for video summary generation.

[0052] (4) Self-attention mechanism The self-attention mechanism is used to model the global dependencies between video frames and effectively capture the contextual information in long-time videos. , the attention weight matrix is ​​calculated by the self-attention mechanism, that is, the self-attention mechanism first maps it to the query ( ),key( ) and value ( The matrix is: ; in, To learn the parameters.

[0053] Then, the attention weight is obtained by calculating the correlation between the query and the key: ; in, is the attention weight matrix, softmax The operation ensures that the weight of each row sums to 1. It is a normalization factor used to alleviate the problem of too large dot product value.

[0054] Then, using the attention weight matrix Pair value vector Perform weighted summation to obtain the updated feature representation, which is: ; The updated features obtained in the above way The global context information is integrated and the important relationships between each time step are encoded into the features, thereby enhancing the model's ability to understand the global features of the video. Through the self-attention mechanism, the model can focus on the key relationships between frames in the video, effectively enhancing the ability to model the global context and providing important support for video summary generation.

[0055] (5) Spatiotemporal Attention Mechanism In order to integrate the information of temporal attention and spatial attention, this embodiment adopts a weighted fusion strategy to combine the two attention distributions to generate the final attention feature. This fusion strategy not only retains the temporal dynamic information of the video, but also enhances the attention to the salient areas.

[0056] Specifically, the updated feature representation is input as the input feature into the spatiotemporal attention layer. The temporal and spatial attention weight matrices are first calculated through the temporal and spatial attention mechanisms, respectively, and are expressed as and , based on the two attention weight matrices, a dynamic weight adjustment mechanism is used to perform weighted fusion of features to obtain key spatiotemporal features. The fused feature representation can be calculated by the following formula: ; in, and It is a hyperparameter, and the ratio of the two can be adjusted according to the specific task scenario; Represents the input features.

[0057] In order to further improve the flexibility and adaptability of the attention mechanism, this embodiment also adopts a dynamic weight adjustment mechanism, which is: First, the scaling factor ( Scaling Factor ), the calculation formula is: ; Secondly, according to the scaling factor, the weight of the feature weighted fusion is adaptively adjusted. Specifically, according to the scaling factor (abbreviated as SF ), the factor SF is used to adjust the original time weight and spatial weight ,like or , so that the relative importance between the two can be adaptively adjusted according to the spatiotemporal structure of the video, thereby balancing the importance of time and space.

[0058] Through the above dynamic adjustment mechanism, when there are significant differences in the number of temporal frames (T) and spatial dimensions (D) of the video, the weights will be adaptively adjusted to balance the importance of the two. This dynamic adjustment mechanism is particularly prominent when the video is long or the feature dimension is large.

[0059] Through the above process, key spatiotemporal features containing key video information can be extracted from the original video frames. This feature is extracted and enhanced through various attention mechanisms such as channel attention, self-attention, and spatiotemporal attention. It can effectively capture the interaction of temporal dynamics and spatial information, highlight the key information in the video, and enhance the feature expression ability.

[0060] (6) Feature Mapping and Skip Connections In the OCVF model, Value Embedding is used as a feature mapping method to express and optimize features. Specifically, the fully connected layer (which can be regarded as part of the MLP) is used to embed the input features. x Nonlinear mapping can generate more recognizable and complex feature representations. The process can be described as: ; in, ReLU is the activation function, and are the weights and biases of the MLP.

[0061] In this embodiment, feature mapping is performed in combination with the above value embedding operation. Figure 4 As shown in Figure 1, first, the extracted key spatiotemporal features (key) are fused with features at different levels through different types of jump connections (such as KC, CF, IF), so that features from different sources complement each other and enhance the understanding of video content. x_att and key Add to enhance the representation ability of features; CF (ConnectionFusion, current feature connection) refers to the feature x_out and x_att Add together to make the final feature contain more semantic information; IF (Input Fusion, intermediate feature connection) refers to adding features x_out With initial features x Fusion is performed to retain the original feature information and enhance the robustness of the model. Through the above jump connection operation, the final feature y , for use in subsequent video summary generation tasks.

[0062] Secondly, the features obtained by skip connection y And the features obtained by nonlinear mapping Fusion, get the fusion feature Z, which can be expressed as: y ; Among them, ⊕ represents the feature fusion operation.

[0063] The final fusion features are generated by the above feature mapping and jump connection, which can enhance the model's feature expression ability, improve the fluidity of information, avoid gradient disappearance, and promote information fusion through the connection between different feature layers. Combining the above feature mapping and fusion, the model can more effectively capture the complex relationships in the input data, thereby enhancing the overall feature expression ability and modeling performance.

[0064] Furthermore, the generated fusion feature Z is subjected to multiple nonlinear mapping and optimization to generate a more representative, recognizable and distinguishable embedding representation, so as to further improve the modeling ability of video temporal and spatial information, and finally provide stronger feature support for subsequent video summary generation tasks. The process is as follows: ; in, and are weights and biases.

[0065] Through the above mapping operation, the model's ability to capture the relationship between different features and the ability to express video content can be improved. The OCVF model can effectively characterize the key features in the input data and achieve significant performance improvement in comprehensive modeling. Moreover, this judgment and mapping operation is completed directly in the forward propagation, avoiding redundant auxiliary function calls, making the entire process more efficient and clear, and providing a good foundation for subsequent expansion to deeper nonlinear mapping. In addition, the combination with other modules further enhances the model's modeling ability for spatiotemporal features. That is, the feature mapping operation plays a core role in the OCVF model, enabling the model to better understand and express video content.

[0066] (7) Classifier In the OCVF model, the classifier optimizes the importance scoring process of video frames through a carefully designed layer structure and activation function. The classifier consists of two key parts: linear transformation and nonlinear activation, and layer normalization is used to enhance the stability and efficiency of the model.

[0067] In the classifier, the input features are first projected into an intermediate feature space through a fully connected linear layer to capture key information. This process can be expressed as: ; in, and are the weights and biases of this layer.

[0068] Then, the ReLU activation function is used to introduce nonlinearity, which helps to enhance the model's ability to learn complex relationships, which can be expressed as: ; The above ReLU activation function helps to prevent the activation value from being too large while keeping the gradient flowing, thereby maintaining the stability of the network.

[0069] Furthermore, in order to control overfitting and improve the generalization ability of the model, the Dropout layer is applied to randomly discard some activation values, which can be expressed as: ; in, Indicates that there is a 50% probability of discarding the output of a neuron.

[0070] Afterwards, the second fully connected layer converts the processed features into the final classification score, which is mapped to the [0, 1] interval through the Sigmoid function, indicating the importance of each video frame, which can be expressed as: ; Among them, σ represents Sigmoid function, and are the weights and biases of the layer.

[0071] Finally, the classifier outputs the predicted frame importance score ,Should is used to calculate the loss function, and the true label is In this embodiment, the loss function includes two parts: classification loss and regression loss, where the classification loss is expressed using cross entropy loss as: ; The regression loss is expressed using the mean square error (MSE) as: ; The total loss is weighted combination: ; in, and is a weight parameter that balances classification and regression losses. This loss function optimizes the classifier’s ability to score frame importance while improving the overall performance of the model. N Indicates the frame number.

[0072] In addition, in order to ensure the numerical stability of the output of each layer of the network, the layer normalization technique is used to standardize the output of each neuron, which can be expressed as: ; Among them, μ and σ represent the mean and standard deviation of the output of the current layer, respectively, and and are learnable scaling and translation parameters.

[0073] Through the above design, the classifier can not only effectively evaluate the importance of each frame of video, but also ensure the effective transmission and integration of features in the network through the integration of layer normalization and skip connections, accelerate the convergence speed of the model, and improve the stability of training.

[0074] Furthermore, the correlation coefficient is used: Kendall's Tau ( τ ) and Spearman's Rank Correlation Coefficient ( ρ ) to evaluate the model to measure the similarity between the summary generated by the model and the summary annotated by the human. The two correlation coefficients are used to evaluate the ranking consistency of the generated summary and the user annotation, thereby verifying the effectiveness and reliability of the model in capturing important frames in the video. By calculating the above correlation coefficients, we can fully understand the performance of the model in ranking consistency and ensure the scientificity and verifiability of the summary quality.

[0075] The following experiments are conducted to further verify the effectiveness of the method proposed in this embodiment. Specifically, the experiment is conducted on a server equipped with an NVIDIA RTX 3090 GPU, and the model training uses the following parameter settings: epochs (rounds) is 100, batch_size (data size) is set to 1, learning_rate (learning rate) is 1e-3, and weight_decay (weight decay) is 1e-7. The selection of these parameters is intended to ensure that the model can converge efficiently in the video summary generation task and avoid overfitting while maintaining model performance; the experimental framework is implemented using PyTorch, combined with GPU acceleration, to ensure the efficiency and stability of the training process.

[0076] In the experiment, two standard video summary datasets SumMe and TVSum were used to verify the performance of the OCVF model. The TVSum dataset is designed for video summarization research and contains 50 videos selected from YouTube. These videos cover 10 different categories, including news reports, documentaries, and educational videos. The average length of the videos is about 5 minutes. Each video has an average of 20 independent annotations, totaling 1,000 annotations. These annotations score the importance of each shot in the video, reflecting the importance of the video clip to the overall content. The SumMe dataset contains 25 videos, covering a variety of topics including holiday travel, sports activities, and personal activities. The video length varies from 1 minute to 6 minutes. SumMe uses a manual annotation method, but it provides video summaries rather than importance scores for individual frames or clips. Each video has at least 15 different manually generated summaries, with a total of 390 summary annotations. These summaries are created by different annotators and reflect the content in the video that is considered to be the most important and representative. While SumMe’s data provides valuable insights into the level of detail of manual summaries, it is limited by the small number of videos and annotations, which may affect its applicability in a wider range of scenarios and the effectiveness of algorithm training.

[0077] The above two datasets are widely used in video summarization tasks and are standard test sets in the current field. By training and evaluating on these two datasets, we can fully verify the performance and adaptability of the OCVF model in different types of videos.

[0078] In order to evaluate the performance of the OCVF model in the task of video summary generation, we selected several existing models such as HSA-RNN, VJMHT, RR-STG, MSVA, SSPVS, GoogleNet, CSTA and DMASum (the above network models are all existing models and will not be repeated here) for specific comparison. These models cover the range from traditional feature extraction methods to the latest technologies based on deep learning, representing the development direction of the field of video summarization.

[0079] Compared with HSA-RNN, OCVF has more outstanding capabilities in feature modeling. HSA-RNN can capture the temporal characteristics of videos to a certain extent by combining multimodal features and recurrent neural networks. However, recurrent neural networks are easily limited by the gradient vanishing problem when dealing with long temporal relationships, and the capture of dynamic features between frames is relatively rough. OCVF not only enhances the modeling ability of the temporal relationship of video frames through improved feature mapping operations, but also improves the correlation expression between multimodal features, thereby showing better performance in video summarization tasks.

[0080] VJMHT's modeling method based on hidden Markov trees is suitable for processing video summarization of structured scenes. Its advantage is that it can model the hierarchical characteristics of videos through a tree structure. However, this method has limited ability to capture global contextual relationships in complex dynamic scenes. OCVF can accurately capture key frames in complex dynamic videos and effectively integrate key information from different modes in the video through comprehensive modeling of spatiotemporal features, thus outperforming VJMHT in complex scenes.

[0081] RR-STG uses spatiotemporal graph modeling to explicitly describe the temporal dynamic information of videos. Its method has a strong advantage in preserving the temporal relationship of videos. However, the high-dimensional characteristics of spatiotemporal graphs limit its performance when processing large-scale video data. OCVF effectively solves the efficiency problems in feature extraction and temporal modeling by introducing channel attention mechanism and flexible jump connection strategy, and has higher computational efficiency and scalability while ensuring performance.

[0082] MSVA uses a multimodal attention mechanism to enhance the feature extraction capability of video content by focusing on the association between multiple modalities. However, its feature expression for non-salient frames is weak, and it is easy to ignore key dynamic changes in the video in complex scenes. OCVF adaptively adjusts the importance of different channels through the channel attention mechanism, so that the model can highlight important dynamic features in the video, especially in diverse video scenes.

[0083] SSPVS is a video summarization method that uses saliency and structural features. Although it performs well in static scenes, it has limited effect when dealing with dynamic and complex scenes. In contrast, OCVF achieves more refined feature capture in dynamic scenes by optimizing feature mapping operations and attention mechanisms, significantly improving the quality of video summarization.

[0084] GoogleNet is a classic convolutional neural network with strong feature extraction capabilities, but its design is not optimized for the spatiotemporal characteristics of video data. OCVF improves the GoogleNet architecture and introduces a channel attention module after each Inception module, significantly enhancing the modeling capabilities of video spatiotemporal features, thereby achieving performance excellence in video summarization tasks.

[0085] CSTA improves the ability to understand video content by combining the spatiotemporal attention mechanism, but its feature expression is less granular, especially in the interactive modeling of cross-modal features. OCVF inherits the spatiotemporal modeling advantages of the CSTA framework and further improves the model's performance in capturing the correlation of multimodal features through channel attention and numerical embedding modules.

[0086] DMASum is based on the extraction and aggregation of deep features and can perform well in specific scenarios, but its generalization ability in cross-domain video tasks is limited. OCVF emphasizes the in-depth modeling of visual features in its design and enhances the understanding of temporal and spatial features through feature mapping operations, thus showing stronger robustness and adaptability in the summary generation of diverse video types.

[0087] In order to verify the effectiveness of OCVF in the video summary generation task, before channel attention is used in the video summary model, it is compared with existing methods, such as HSA-RNN, VJMHT, RR-STG, MSVA, SSPVS, GoogleNet, CSTA, and DMASum as comparison models. The experimental results are shown in Table 1. The experimental results show that OCTV has a significant advantage on the SumMe dataset, achieving the current best performance, and also maintains a high score on the TVSum dataset, reflecting its stability and generalization ability.

[0088] Table 1: Comparison of results of different models

[0089] Among them, τ and ρ are two correlation coefficients for measuring ranking consistency. τ is used to compare the pairwise consistency between two ranking sequences and is suitable for measuring the stability of rankings. ρ is calculated based on the rank values ​​of the two variables, reflecting the overall consistency between the rankings and is suitable for evaluating the strength of the correlation between variables.

[0090] In contrast, although DMASum achieved good performance on the TVSum dataset, its score on the SumMe dataset was significantly lower, showing that its adaptability to different datasets is limited. OCVF, by integrating channel attention mechanisms and feature mapping operations, is more comprehensive and efficient in video feature extraction and temporal modeling, and can simultaneously extract key frames in dynamic and complex scenes. In particular, OCVF's balanced performance on the two datasets demonstrates its wide applicability to different video types and content.

[0091] In addition, the design of OCVF emphasizes the in-depth modeling of visual dependencies. Compared with traditional methods that only focus on temporal dependencies, OCVF adds a channel attention mechanism to each feature module, which can not only capture key visual background information, but also enhance the understanding of the scene through the absolute position information of the frame. This ability enables OCVF to excel in capturing key moments and generating highly generalized summaries, especially in the SumMe dataset, where OCVF's performance further verifies its robustness in complex and diverse videos.

[0092] In addition, OCVF also has the advantages of high efficiency and simplicity. Unlike methods that require additional modal data or specific task features, OCVF learns the feature relationship between frames through a deep convolutional network, combined with feature mapping operations and attention strategies, to achieve higher performance at a lower computational cost. This design enables OCVF to outperform existing methods in terms of τ and ρ indicators, and demonstrate its unique advantages and innovations in the task of video summary generation.

[0093] In addition, an ablation experiment was designed in this experiment to verify the contribution of the main modules of the model (method) to the overall performance. The channel attention mechanism and feature mapping operation were removed respectively, where the feature mapping operation specifically refers to the value embedding operation, and the performance on the SumMe and TVSum datasets was compared. The experimental results are shown in Table 2 below.

[0094] Table 2: Ablation experiment results

[0095] Specifically, after removing the channel attention mechanism, the τ and ρ values ​​of the model on the SumMe dataset dropped from 0.261 and 0.290 to 0.249 and 0.278 respectively; on the TVSum dataset, the τ and ρ values ​​dropped from 0.198 and 0.260 to 0.191 and 0.251 respectively. This shows that the channel attention mechanism plays a vital role in the cross-channel fusion of features and can better capture the complex correlation information in video data.

[0096] In addition, the improved feature mapping operation plays an important role in the model. On the SumMe dataset, the τ and ρ values ​​are 0.246 and 0.275 respectively; on the TVSum dataset, the τ and ρ values ​​are 0.195 and 0.256 respectively, which shows that the feature mapping operation has a positive impact on the model performance by enhancing the feature expression ability and effectively enhances the feature expression ability.

[0097] Through the above ablation experiments, it can be concluded that the feature mapping operation plays an important role in improving the feature expression ability, while other modules provide additional support for the performance optimization of the model. Feature mapping effectively transforms the input data through multi-layer nonlinear mapping and feature fusion operations, thereby enhancing the model's ability to express key features. Compared with traditional feature processing methods, the improved feature mapping operation pays more attention to fine-grained feature expression and can more accurately capture the changes in spatiotemporal information and key dynamic features in the video. Experimental results show that OCVF with improved feature mapping operations has achieved significant performance improvements on both SumMe and TVSum datasets. When identifying dynamic scenes and key objects, the improved feature mapping enables the model to better focus on important features, thereby improving the accuracy and overall quality of video summary generation. Compared with traditional models, it shows stronger robustness and detail capture capabilities, especially in diverse and complex scenes.

[0098] The channel attention mechanism plays a core role in capturing the correlation of multi-channel features, while the feature mapping operation provides additional support for feature representation. The channel attention module counts the features of each channel through global average pooling, and generates channel weights through two fully connected layers (ReLU and Sigmoid activation functions), thereby achieving channel-by-channel weight adjustment. This module enhances the model's response to salient features, suppresses unimportant features, and significantly improves the effectiveness of feature expression. Compared with spatial attention or global attention, channel attention pays more attention to fine-grained channel-level feature selection and can better model the dynamic changes of spatiotemporal information in videos. Experiments show that OCVF with the addition of the channel attention module achieves significant performance improvements on the SumMe and TVSum datasets. When identifying dynamic scenes and important objects, the channel attention mechanism enables the model to focus more on significantly changing features, thereby improving the accuracy and quality of video summary generation. Compared with traditional models, OCVF shows stronger performance in Kendall's Tau (τ) and Spearman's Rank Correlation (ρ) indicators, especially in diverse and complex scenarios, showing stronger generalization and detail capture capabilities. The complete OCVF model achieves optimal performance by integrating these modules.

[0099] In summary, the new video summary model based on OCVF proposed in this embodiment solves the problems of insufficient key feature extraction in multimodal videos and the understanding of the spatiotemporal relationship of video features in existing methods. OCVF can adaptively adjust the feature channel weights by introducing a channel attention mechanism, and combines a variety of position encoding strategies and feature mapping operations, which significantly improves the model's understanding of video content and feature synthesis capabilities. Experimental results on standard datasets SumMe and TVSum show that OCVF has significantly improved performance compared to current mainstream methods, reflecting its robustness and wide applicability in complex scenes and diverse video types; the OCVF model provides an effective and efficient solution for video summary generation, and its innovative design is not only of great theoretical significance, but also provides a new direction for the further development of video understanding and generation technology.

[0100] Embodiment 2 This embodiment provides a video summary generation system based on feature expression enhancement, including: A video acquisition module is used to acquire the original video and extract the video frame image sequence; The video summary generation module is used to input the video frame image sequence into the OCVF video understanding model. Each input video frame image is first extracted with initial features by the feature extraction module, and the initial features are extracted with key spatiotemporal features by the attention module. Then, high-level feature representation is generated through feature mapping and jump connection. Finally, the high-level feature representation outputs the importance score of each video frame through the classifier, determines the key frame, and generates the final video summary result. Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer which are arranged in sequence; the initial features are adaptively adjusted through the channel weights of the channel attention layer to generate key features; after the dimensionality reduction processing through the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted through the self-attention layer and the spatiotemporal attention layer.

[0101] Embodiment 3 This embodiment provides an electronic device, including: a memory, used to store executable instructions; and a processor, used to implement the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0102] Embodiment 4 This embodiment also provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above method provided in this embodiment.

[0103] Embodiment 5 This embodiment provides a computer program product, which includes an executable instruction, which is a computer instruction; the executable instruction is stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instruction from the computer-readable storage medium and the processor executes the executable instruction, the electronic device executes the above method provided in this embodiment.

[0104] The steps involved in the above embodiments 2 to 5 correspond to the method embodiment 1. For the specific implementation, please refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0105] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0106] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A video summary generation method based on feature expression enhancement, characterized in that: include: Get the original video and extract the video frame image sequence; The video frame image sequence is input into the OCVF video understanding model. Each input video frame image is first extracted with initial features by the feature extraction module. The initial features are then extracted with key spatiotemporal features by the attention module. Then, high-level feature representation is generated through feature mapping and skip connection. Finally, the high-level feature representation outputs the importance score of each video frame through the classifier, determines the key frames, and generates the final video summary result. Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer which are arranged in sequence; the initial features are adaptively adjusted through the channel weights of the channel attention layer to generate key features; after the dimensionality reduction processing through the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted through the self-attention layer and the spatiotemporal attention layer.

2. The method for generating a video summary based on feature expression enhancement according to claim 1, characterized in that: In the channel attention layer, the input initial feature map is subjected to global average pooling to compress the spatial information to obtain a global description of each channel, and then the channel attention information is learned through convolution, and the attention weight of each channel is obtained using the activation function; Based on the attention weight of each channel, the original initial feature map of the input is multiplied element by element to obtain the channel-weighted feature map, i.e., the key feature map.

3. The method for generating a video summary based on feature expression enhancement according to claim 1, characterized in that: In the position coding layer, for the feature representation of each input video frame image, the position information of the current video frame in the original video is introduced to obtain the feature representation after position coding; The position information is represented by a position coding matrix, which is calculated by using any one or more methods of fixed position coding, relative position coding, and learning position coding.

4. The method for generating a video summary based on feature expression enhancement according to claim 1, characterized in that: The position-encoded features are sequentially passed through the self-attention layer and the spatiotemporal attention layer to extract key spatiotemporal features, including: The position-encoded features are input as input features to the self-attention layer, and the attention weight matrix is ​​calculated through the self-attention mechanism. The updated feature representation is obtained by weighted summation of the attention weight matrix. The updated feature representation is then input into the spatiotemporal attention layer as input features. The temporal and spatial attention weight matrices are calculated through the temporal and spatial attention mechanisms respectively. Based on the two attention weight matrices and using a dynamic weight adjustment mechanism, feature weighted fusion is performed to obtain the key spatiotemporal features.

5. The method for generating a video summary based on feature expression enhancement according to claim 4, characterized in that: The dynamic weight adjustment mechanism is: The scaling factor is calculated based on the number of time frames and spatial dimensions of the original video; According to the scaling factor, the weights of feature weighted fusion are adaptively adjusted to balance the importance of time and space.

6. The method for generating a video summary based on feature expression enhancement according to claim 1, characterized in that: The feature map and skip connection are: Use multi-layer perceptron to perform nonlinear mapping on input features and generate complex feature representations; Through skip connections, the complex feature representation is fused with the extracted initial features, multi-level intermediate features, and key spatiotemporal features to obtain fused features; wherein the multi-level intermediate features include key features; The fused features are then subjected to multiple nonlinear mappings to generate the final embedded representation, i.e., the high-level feature representation.

7. A video summary generation system based on feature expression enhancement, characterized in that: include: A video acquisition module is used to acquire the original video and extract the video frame image sequence; The video summary generation module is used to input the video frame image sequence into the OCVF video understanding model. Each input video frame image is first extracted with initial features by the feature extraction module, and the initial features are extracted with key spatiotemporal features by the attention module. Then, high-level feature representation is generated through feature mapping and jump connection. Finally, the high-level feature representation outputs the importance score of each video frame through the classifier, determines the key frame, and generates the final video summary result. Among them, the attention module includes a channel attention layer, an average pooling layer, a position encoding layer, a self-attention layer and a spatiotemporal attention layer which are arranged in sequence; the initial features are adaptively adjusted through the channel weights of the channel attention layer to generate key features; after the dimensionality reduction processing through the average pooling layer, the temporal and spatial position information of the video frame is embedded into the low-dimensional features through the position encoding layer; finally, the key spatiotemporal features are extracted through the self-attention layer and the spatiotemporal attention layer.

8. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is configured to implement the method for generating a video summary based on feature expression enhancement according to any one of claims 1 to 6 when executing the executable instructions stored in the memory.

9. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute the executable instructions to implement a video summary generation method based on feature expression enhancement as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the method for generating a video summary based on feature expression enhancement according to any one of claims 1 to 6 is implemented.

Images