Video abstraction method fusing cross-modal semantic information

Abstract

The invention discloses a video abstraction method fusing cross-modal semantic information, and belongs to the technical field of computer vision. The method comprises the following steps: firstly, extracting an image frame sequence and a motion frame sequence from an input video, and then respectively extracting static features and dynamic features by using a cross-modal feature extraction network; then, frame features are processed through a space-time convolution association attention mechanism, an attention graph reflecting the space-time importance of the frame features is generated, and meanwhile intra-frame space information and inter-frame time information are captured; in addition, a cross-modal dynamic fusion module and a semantic consistency corrector are introduced to optimize a video abstract generation process, reduce noise interference and improve abstract quality. And finally, constructing an objective function, training a video abstract generation model through unsupervised or supervised learning, and generating a dynamic video abstract according to the predicted importance score. According to the method, the static and dynamic characteristics in the video are comprehensively utilized, and the semantic accuracy and content coherence of the abstract are improved.

Description

Technical Field

[0001] The invention relates to a video summarization method for fusing cross-modal semantic information, and belongs to the technical field of image communication. Background Art

[0002] With the rapid development of the Internet and social media, the amount of video data is growing exponentially. How to efficiently extract key information from massive video data and generate concise and representative video summaries has become an important topic in current research. Video summarization not only has important application value in the retrieval, management and display of video content, but also plays an indispensable role in automated editing, intelligent monitoring and other fields. However, existing video summarization methods still have many limitations, especially in dealing with dynamic scenes and capturing motion information in videos.

[0003] Traditional video summarization techniques can be roughly divided into two categories: key frame selection based on static images and dynamic event recognition based on video content. Key frame extraction methods based on static images usually rely on image features such as color, texture, and shape to select representative images as summaries of videos. However, this method ignores the motion information in the video and cannot fully reflect the dynamic changes in the video content. Although the dynamic event recognition method based on video content can take motion information into account, it is often limited by time and space characteristics when dealing with complex scenes, resulting in the generated summary not being able to effectively reflect the important events or plots in the video.

[0004] In the process of generating video summaries, capturing motion information is crucial. Motion information in videos (such as the motion trajectory of objects, dynamic changes in scenes, etc.) is often a key factor in expressing the core content of the video. However, traditional methods do not fully consider the dynamic evolution of motion information, resulting in the lack of sufficient details and accuracy in the generated summaries when expressing the video plot.

[0005] In recent years, with the rapid development of deep learning technology, spatiotemporal flow networks, as an emerging model structure, have shown significant advantages in processing video data. Spatiotemporal flow networks can simultaneously capture the spatial characteristics and temporal dynamic changes of videos, and have achieved excellent results in the fields of action recognition and video analysis. Through spatiotemporal flow networks, the motion information in the video can be modeled more accurately, so that the video summary can more comprehensively reflect the key events and dynamic content in the video.

[0006] However, existing spatio-temporal flow network methods usually focus on specific tasks (such as action recognition, video classification, etc.) and are not specifically optimized for video summary generation. Traditional video summary methods fail to fully combine the motion information features in the spatio-temporal flow network, resulting in a large room for improvement in summary quality. Therefore, how to effectively integrate the spatio-temporal flow network with motion information to improve the quality of video summaries has become an urgent problem to be solved in current research. Summary of the Invention

[0007] The purpose of the present invention is to provide a video summary method that fuses cross-modal semantic information, aiming to solve the problem that the existing technology does not make full use of static and motion features in videos, resulting in the model being unable to comprehensively and accurately understand video content.

[0008] To achieve the above objective, the present invention provides a video summary method that fuses cross-modal semantic information, including the following steps: S1: Read the input video, extract the image frame sequence for representing static visual content, and the motion frame sequence reflecting the dynamic motion state changes; S2: Respectively extract the static features and dynamic features of the video frames through a cross-modal feature extraction network. The cross-modal feature extraction network includes a temporal flow network and a spatial flow network. The spatial flow network is used to extract static features, and the temporal flow network is used to extract dynamic features; S3: Generate a spatio-temporal importance attention map through a spatio-temporal convolutional correlation attention mechanism; S4: Introduce a cross-modal dynamic fusion module to dynamically adjust the weight ratio of static and dynamic modalities according to the semantic features of the current frame, and generate a hybrid feature representation that fuses cross-modal semantics; S5: Introduce a semantic consistency corrector to optimize the semantic consistency between cross-modal features and static features; S6: Construct an objective function, train the video summary generation model, and generate a video summary according to the importance scores predicted by the model.

[0009] Further, the spatial flow network in the video summary method that fuses cross-modal semantic information is used to extract static features reflecting the object categories, scene semantics, and visual content in the video frames.

[0010] Further, the temporal flow network in the video summary method that fuses cross-modal semantic information is used to extract two dynamic features, namely motion RGB features and optical flow features. The motion RGB features are used to capture scene switching and color changes of dynamic objects, and the optical flow features are used to describe the inter-frame motion direction and speed. By simultaneously extracting the two dynamic features, the model can more comprehensively capture the motion information in the video and effectively distinguish different types of dynamic changes.

[0011] Furthermore, in the video summarization method that fuses cross-modal semantic information, the spatio-temporal convolutional correlation attention mechanism first stacks the video frame features to form a frame feature representation with a two-dimensional structure; then it uses a convolutional neural network to generate an attention map reflecting the spatio-temporal importance of the frame features, while capturing the spatial information within the frame and the temporal information between frames.

[0012] Furthermore, the cross-modal dynamic fusion module in the video summarization method that fuses cross-modal semantic information is a key improvement introduced into the spatio-temporal convolutional correlation attention mechanism. It dynamically calculates the ratio of static and dynamic modality weights (α and β) according to the features of the current frame through a small multi-layer perceptron. This module receives the embedded frame features as input, and outputs the normalized weight ratio after being processed by the multi-layer perceptron, where α + β = 1. In this way, the model can adaptively adjust the allocation of static and dynamic modality weights according to the video content, so as to more flexibly process different types of video content and enhance the ability to identify key frames.

[0013] Furthermore, the semantic consistency corrector in the video summarization method that fuses cross-modal semantic information can dynamically adjust the semantic matching degree between cross-modal features and video static features, effectively reducing the noise caused by motion features, thereby enhancing the semantic accuracy and content coherence of the generated video summary.

[0014] Furthermore, the objective function in the video summarization method that fuses cross-modal semantic information includes the following terms The objective function is defined as: , where: is the reward function term, which is used to evaluate the importance and coverage of the frames in the generated summary; is the regularization term, which is used to prevent the model from overfitting; is the semantic consistency loss term, which is used to optimize the semantic consistency between cross-modal features and appearance features; and are hyperparameters, which are used to balance the weights of different loss terms.

[0015] The reward function term is used to evaluate the similarity between the generated summary and the annotated summary, so as to measure the quality of the summary generated by the model. The specific calculation formula is as follows: , where: is the total number of video frames; is the annotated importance score of the th frame; is the prediction of the model for the The importance score of a frame, ranging between [0, 1].

[0016] The regularization term is used to prevent the model from overfitting, and the weight decay method is adopted. The specific calculation formula of the regularization term of weight decay is as follows: , where represents all the parameters of the model, represents the L2 norm of the parameters. By adding the weight decay term, the scale of the model parameters can be restricted, thereby improving the generalization ability of the model.

[0017] The semantic consistency loss term is used to optimize the semantic consistency between cross-modal features and appearance features, and reduce the noise interference in motion features. The specific calculation formula is as follows: , where, is the hybrid feature vector after fusing temporal and spatial features, is the static feature vector extracted by the temporal flow, N is the total number of video frames; represents the Euclidean distance of the vectors.

[0018] Furthermore, the training steps of the model proposed in the method for enhancing video summary of motion information based on spatio-temporal flow network include: constructing a video summary generation network; constructing a training set including original video frames and their corresponding annotated importance scores; inputting the training set into the network for training; the network outputs the predicted importance scores of the frames; calculating the loss between the predicted scores and the annotated scores; when the loss is minimized, the model converges, and stop training to obtain the trained network.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention not only considers the static visual content in the video, but also comprehensively analyzes the dynamic motion state, so that the video summary can more comprehensively reflect the video content.

[0020] 2. By extracting and utilizing motion RGB features and optical flow features, the present invention can more accurately capture important dynamic events in the video and improve the accuracy of the summary.

[0021] 3. The spatio-temporal convolutional correlation attention mechanism proposed by the present invention enhances the spatio-temporal importance of frame features, improves the ability to capture key information in the video summary, and reduces the computational cost at the same time.

[0022] 4. The cross-modal dynamic fusion module introduced by the present invention enables the model to adaptively balance static and dynamic features, optimizes the feature fusion process, and improves the accuracy of the summary.

[0023] 5. The semantic consistency corrector proposed by the present invention enhances the semantic accuracy and content coherence of video summaries by reducing noise interference. Description of the Drawings

[0024] Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 It is a flowchart of the implementation of the video summary method for fusing cross-modal semantic information provided by the present invention; Figure 2 It is a network framework of the video summary method for fusing cross-modal semantic information provided by an embodiment of the present invention; Figure 3 It is a schematic structural diagram of the spatio-temporal convolutional correlation attention mechanism provided by an embodiment of the present invention; Figure 4 It is a schematic structural diagram of the cross-modal dynamic fusion module provided by an embodiment of the present invention; Figure 5 It is a schematic structural diagram of the semantic consistency corrector provided by an embodiment of the present invention. Detailed Embodiments

[0025] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several modifications and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0026] As Figure 1 shown, it is a flowchart of the implementation of the video summary method for fusing cross-modal semantic information provided by the present invention, including the following steps: S1. Read the input video and extract the image frame sequence and the motion frame sequence; S2. Extract the static features and dynamic features of the video frames through the cross-modal feature extraction network; S3. Generate the spatio-temporal importance attention map through the spatio-temporal convolutional correlation attention mechanism; S4. Introduce the cross-modal dynamic fusion module into the spatio-temporal convolutional correlation attention mechanism; S5. Introduce the semantic consistency corrector to optimize the video summary generation process; S6. Construct the objective function, train the video summary model, predict the importance of new video frames, and generate the summary.

[0027] Embodiment 1: The present invention provides a preferred embodiment to execute S1. First, read the input video and obtain two types of frame sequences: one is an image frame sequence reflecting static visual content, and the other is a motion frame sequence reflecting the change of the dynamic motion state of the video. This method is applicable to processing various types and durations of video content, such as movies, surveillance videos, sports event videos, etc., and can efficiently process both long videos and short videos. The specific operation steps are as follows: Extract the video frame sequence by means of uniform sampling. First, select two frames per second as the sampling rate, which means that for a video, one frame is extracted every 0.5 seconds. Then, according to the sampling rate, frames are uniformly extracted from the beginning to the end of the video to form a frame sequence , where T is the total number of frames extracted, represents the t th frame at the time point. Finally, the extracted frame sequence is fed into a pre-trained neural network model to extract static features and dynamic features.

[0028] Embodiment 2: The present invention provides a preferred embodiment to execute S2. Use the neural network model to extract features from the image frame sequence and the motion frame sequence to obtain frame-level static features and frame-level motion features. As Figure 2 shown, it is the network framework of the video summarization method for fusing cross-modal semantic information provided by the embodiment of the present invention. The specific operation steps are as follows: S21, extract static features. The spatial stream network focuses on extracting static features in video frames. Static features are mainly used to capture the object category, scene semantics, and semantic information of visual content in video frames. First, input the sampled frames into a pre-trained deep convolutional neural network model. Here, GoogleNet is used as the basic model of the spatial stream network. Then, extract the feature vectors of the pool5 layer with a dimension of 1024 in the model. These feature vectors reflect the object category, scene semantics, and semantic information of visual content in the frames, and can help the model understand the static content within the frames.

[0029] S22, extract dynamic features. The temporal stream network focuses on extracting dynamic features in video frames, including motion RGB features and optical flow features. First, input the sampled frames into a pre-trained I3D model. The I3D model is pre-trained on large-scale video datasets such as Kinetics and can effectively capture the dynamic information in the video. Then, extract two types of motion features from the model: motion RGB features X r and optical flow features X f .

[0030] Motion RGB features can capture color changes and scene transition information in video frames. These features reflect visual changes in the video, such as scene transitions or color changes of objects. First, the obtained frame sequence is input into the RGB stream of the I3D model for processing; then, the Mixed_5c layer is selected to extract feature points; and then, the extracted feature vectors are normalized to ensure consistent numerical ranges.

[0031] Optical flow features focus on pixel motion between frames and can directly reflect motion direction and speed information in the video, such as a person walking, an object moving, or dynamic changes in the background. First, an optical flow field is calculated in the input video , where each o t is the optical flow map between the t-th frame and the (t + 1)-th frame. The calculated optical flow field sequence O is input into the optical flow stream of the I3D model. The optical flow stream is specifically designed to process optical flow information and can capture motion patterns in the video.

[0032] Embodiment 3: The present invention provides a preferred embodiment for executing S3. Through a spatio-temporal convolutional correlation attention mechanism, it is used to fuse spatial information within a frame and temporal information between frames to generate an attention map reflecting the spatio-temporal importance of frame features. As Figure 3 shown, it is a schematic structural diagram of the spatio-temporal convolutional correlation attention mechanism provided by the embodiment of the present invention. The specific steps are as follows: First, the features extracted from the spatial stream and the temporal stream are stacked to form a frame feature representation in a two-dimensional structure. The static features of the spatial stream and the dynamic features of the temporal stream are concatenated along the feature dimension to obtain a high-dimensional frame feature representation. Then, a convolutional neural network is used to process the stacked frame features. The convolutional neural network can simultaneously capture spatial information within a frame and temporal information between frames to generate an attention map reflecting the spatio-temporal importance of frame features. Each element of the attention map represents the importance of the corresponding frame feature. The attention map can be used for subsequent feature weighting and importance score prediction.

[0033] Embodiment 4: The present invention provides a preferred embodiment for executing S4. By introducing a cross-modal dynamic fusion module into the spatio-temporal convolutional correlation attention mechanism, the model can dynamically calculate the proportion of static and dynamic modal weights (α and β) according to the features of the current frame. As Figure 4 shown, it is a schematic structural diagram of the cross-modal dynamic fusion module provided by the embodiment of the present invention. The specific steps are as follows: First, a multi-layer perceptron is used to dynamically calculate the proportion of static and dynamic modal weights according to the features of the current frame. The input of this network is the feature representation E of the current frame, and the output is the weight proportions α and β. The specific formula is as follows: , Among them, α and β represent the proportions of the temporal weight and the spatial weight respectively, and satisfy α + β = 1.

[0034] Then, according to the calculated weight proportions, the temporal and spatial weights are adjusted. Assume that the original temporal and spatial weights are and respectively. The adjusted weights are: , The adjusted weights are fused with the frame features to generate a hybrid feature representation M . The specific formula is as follows: , Among them, is the frame feature representation, represents element-wise multiplication.

[0035] Finally, using the hybrid feature representation M , the importance score of each frame is predicted. The hybrid features are mapped to the importance score through a fully connected layer and a Sigmoid activation function, and the score range is between 0 and 1. The specific formula is as follows: , Among them, represents the Sigmoid activation function, is the predicted importance score.

[0036] Embodiment 5: The present invention provides a preferred embodiment to execute S5, designing a semantic consistency corrector to optimize the video summary generation process. By dynamically correcting the semantic consistency between cross-modal features and static features, the noise interference in the motion features is reduced, thereby improving the semantic accuracy and coherence of the video summary. As Figure 5 shown, it is a schematic structural diagram of the semantic consistency corrector provided by the embodiment of the present invention. The specific steps are as follows: S51, semantic correction of cross-modal features and static features: First, calculate the semantic consistency between cross-modal features and static features. Then, the cosine similarity between the two is calculated to measure the semantic consistency. Cosine similarity is a method to measure the similarity of the directions of two vectors and is applicable to high-dimensional feature vectors. The calculation formula is as follows: , Among them, is the hybrid feature vector after fusing temporal and spatial features, is the static feature vector extracted from the time flow, ∙ represents the dot product of vectors, represents the norm of the vector. By calculating the cosine similarity, the semantic consistency between cross-modal features and appearance features can be quantified, providing a basis for the subsequent optimization process.

[0037] S52, Semantic Consistency Loss Calculation: Define a semantic consistency loss function to optimize the semantic consistency of the model. The calculation formula is as follows: , where N is the total number of video frames; represents the Euclidean distance of the vector. The purpose of this loss function is to minimize the difference between cross-modal features and appearance features, thereby improving the model's semantic understanding ability of video content. By optimizing this loss function, the semantic matching degree between cross-modal features and appearance features can be dynamically adjusted, and the noise interference in motion features can be reduced.

[0038] S53, Noise Suppression and Optimization: By minimizing the semantic consistency loss, dynamically adjust the semantic matching degree between cross-modal features and static features, and reduce the noise interference caused by motion features. The optimized model can generate video summaries with more accurate semantics and more coherent content.

[0039] Example Six: The present invention provides a preferred embodiment to execute S6, construct an objective function, and train the video summary generation model in an unsupervised or supervised learning manner. The specific steps are as follows: The objective function includes a reward function term, a regularization term, and a semantic consistency loss term. The reward function term is used to evaluate the importance and coverage of frames in the summary, the regularization term is used to prevent the model from overfitting, and the semantic consistency loss term is used to optimize the semantic consistency between cross-modal features and static features.

[0040] The objective function is defined as: , where: is the reward function term, which is used to evaluate the importance and coverage of frames in the generated summary; is the regularization term, which is used to prevent the model from overfitting; is the semantic consistency loss term, which is used to optimize the semantic consistency between cross-modal features and appearance features (defined in Example Five); and are hyperparameters used to balance the weights of different loss terms.

[0041] The reward function term is used to evaluate the similarity between the generated summary and the annotated summary, thereby measuring the quality of the summary generated by the model. The specific calculation formula is as follows: , where: N is the total number of video frames; is the i annotation importance score of the is the importance score of the i th frame predicted by the model, ranging from [0, 1]. This loss function encourages the model to generate summaries similar to the annotated summaries by minimizing the cross-entropy between the annotated scores and the predicted scores.

[0042] The regularization term is used to prevent the model from overfitting, usually by methods such as weight decay or Dropout. The regularization term for weight decay can be expressed as: , where represents all the parameters of the model, represents the L2 norm of the parameters. By adding the weight decay term, the scale of the model parameters can be restricted, thereby improving the generalization ability of the model.

[0043] The semantic consistency loss term has been defined in Embodiment 5, and its purpose is to reduce the noise interference in the motion features by optimizing the semantic consistency between the cross-modal features and the appearance features.

Claims

1. A video summarization method integrating cross-modal semantic information, characterized in that: The following steps are involved: S1: read the input video, extract the image frame sequence used to represent the static visual content, and the motion frame sequence reflecting the dynamic motion state change; S2: extracting static features and dynamic features of the video frame respectively through a cross-modal feature extraction network, wherein the cross-modal feature extraction network includes a temporal stream network and a spatial stream network, wherein the spatial stream network is used to extract static features, and the temporal stream network is used to extract dynamic features; S3: Generate spatiotemporal importance attention map through spatiotemporal convolutional association attention mechanism; S4: Introduce a cross-modal dynamic fusion module to dynamically adjust the weight ratio of static and dynamic modes according to the semantic features of the current frame, and generate a hybrid feature representation that integrates cross-modal semantics; S5: Introduce a semantic consistency corrector to optimize the semantic consistency between cross-modal features and static features; S6: Construct an objective function, train a video summary generation model, and generate a video summary based on the importance score predicted by the model.

2. The video summarization method for integrating cross-modal semantic information according to claim 1, characterized in that: The spatial stream network is used to extract static features that reflect object categories, scene semantics, and visual content in video frames.

3. The video summarization method for integrating cross-modal semantic information according to claim 1, characterized in that: The temporal flow network is used to extract two dynamic features: motion RGB features and optical flow features. The motion RGB features are used to capture scene switching and color changes of dynamic targets, and the optical flow features are used to describe the direction and speed of motion between frames.

4. The video summarization method for fusing cross-modal semantic information as claimed in claim 1, characterized in that: The spatiotemporal convolutional associated attention mechanism stacks the video frame features to form a frame feature representation with a two-dimensional structure, and uses a convolutional neural network to generate an attention map that reflects the spatiotemporal importance of frame features, while capturing the spatial information within the frame and the temporal information between frames.

5. The video summarization method for integrating cross-modal semantic information as claimed in claim 1, characterized in that: The cross-modal dynamic fusion module dynamically calculates the ratio of static and dynamic modal weights according to the semantic features of the current frame, adjusts the static and dynamic modal weights according to the ratio, and fuses the adjusted weights with the frame features. Generate a hybrid feature representation that integrates cross-modal semantics and then predict the importance score of each frame.

6. The video summarization method for fusing cross-modal semantic information as claimed in claim 1, characterized in that: The semantic consistency corrector dynamically adjusts the semantic matching degree between the cross-modal features and the video static features.

7. The video summarization method for fusing cross-modal semantic information as claimed in claim 1, characterized in that: The objective function is defined as: , in: L reward is a reward function term used to evaluate the importance and coverage of frames in the generated summary; L reg is a regular term used to prevent the model from overfitting; L sem It is a semantic consistency loss term, which is used to optimize the semantic consistency between cross-modal features and appearance features; and is a hyperparameter used to balance the weights of different loss terms; The reward function term is used to evaluate the similarity between the generated summary and the annotated summary, thereby measuring the quality of the summary generated by the model; the specific calculation formula is as follows: ， in: N is the total number of video frames; It is The annotation importance score of the frame; The model predicts The importance score of the frame, ranging from [0, 1]; The regularization term is used to prevent the model from overfitting, usually using methods such as weight decay or Dropout; the regularization term of weight decay can be expressed as: , in represents all the parameters of the model, Represents the L2 norm of the parameter; by adding the weight decay term, the scale of the model parameters can be limited, thereby improving the generalization ability of the model; The semantic consistency loss term is used to optimize the semantic consistency between cross-modal features and appearance features and reduce noise interference in motion features. The specific calculation formula is as follows: , in, It is a mixed feature vector after integrating time and space features. is the static feature vector extracted from the time stream, N is the total number of video frames; Represents the Euclidean distance of vectors.

Images