A method for extracting a multi-modal video summary based on a video caption

Abstract

The application discloses a kind of multi-modal video abstract extraction methods based on video caption, comprising:1 obtains the frame feature representation of video, 2.Obtain the feature representation of caption, 3.Important evaluation of automatic video frame, 5.Optimization abstractor model, 6.Optimization key frame-based video caption generator.The application can quickly output the key frame set of short video and its corresponding caption, wherein the key frame set reflects the overall content of the video in the form of vision with a small number of video frames, and the matching caption summarizes the video picture in the form of text, helping users to more efficiently screen short videos, and can save storage space and computing resources, and is more conducive to deployment applied to terminal equipment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically a method for extracting multimodal video summaries based on video subtitles. Background Technology

[0002] The booming development of short-video social media and self-media has led to an explosive growth in internet video content, making the rapid extraction of key information from videos a crucial issue. The goal of video summarization is to retrieve keyframes or key shots from videos, containing as much information as possible with minimal redundancy. A direct application of video summarization is the display of video thumbnails on video websites; a well-chosen summary helps users decide whether to click on a video. However, the unique characteristics of video summarization tasks, such as the high degree of subjectivity in results, the difficulty of dataset annotation, and variations in video resolution, present significant challenges to improving video summarization technology.

[0003] The aforementioned difficulties in dataset annotation result in a persistent shortage of high-quality datasets in the video summarization field. Previous video summarization methods, such as Xu et al.'s 2022 paper "MHSCNet: A Multimodal Hierarchical Shot-aware Convolutional Network for Video Sum," often rely on the TVSum and SumMe datasets. For instance, the TVSum dataset uses 20 annotators to score the importance of each frame in each video, containing 50 videos. SumMe, on the other hand, uses 15 to 20 annotators to select key segments from only 20 videos. Manual annotation of large-scale video summarization datasets is prohibitively costly and impractical. Previous work has typically used several low-quality datasets as supplementary training. How to train a high-quality video summarization model using existing datasets without incurring additional annotation costs, and how to effectively utilize the summarized video frames, remains a pressing issue. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing a multimodal video summarization method based on video subtitles. This method aims to simultaneously output video summaries and video subtitles, thereby helping users to filter short videos more efficiently, saving storage space and computing resources, and making it more suitable for deployment on terminal devices.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The present invention provides a multimodal video summarization extraction method based on video subtitles, characterized by the following steps:

[0007] Step 1: Obtain the frame feature representation of the video:

[0008] For a video subtitle dataset D = {V, Y}, where V represents the video set and Y represents the set of English subtitles for each video in the video set V;

[0009] A visual encoder using the CLIP model processes any i-th video in the video set V to obtain the frame feature representation F of the i-th video. i ={f i,1, f i,2 ,...,f i,n ,..,f i,N}; where f i,n Let N represent the feature representation of the nth frame in the i-th video, and N represent the total number of frames in video i.

[0010] Step 2: Obtain the feature representation of the subtitles:

[0011] The text encoder using the CLIP model pairs the English subtitle statement Y corresponding to the i-th video. i ={y i,1,1 ,...,y i,1,W ;y i,m,1 ,y i,m,2 ,...,y i,m,t ,...,y i,m,W ;y i,M,1 ,...,y i,M,W The process is performed to obtain the English subtitle text vector T corresponding to video i. i ={t i,1 ,t i,2 ,...,t i,m ,..,t i,M}, where y i,m,t This represents the t-th word in the m-th subtitle statement corresponding to the i-th video, where t i,m This represents the m-th subtitle vector in the corresponding English subtitle text of the i-th video; W represents the total number of words;

[0012] Step 3: Use equation (1) to obtain the feature representation f of the nth frame in the i-th video. i,n With subtitle text vector T i Average similarity s(f i,k ,T i ), and use it as the feature representation of the nth frame of video i. i,n Automated scoring

[0013]

[0014] In equation (1), tr represents the vector transpose;

[0015] Step 4: Construct a video summarizer, including: a self-attention mechanism layer, a local attention enhancement layer, and a fully connected network MLP, and train it;

[0016] Step 4.1: The self-attention mechanism layer uses equation (2) to calculate the feature representation f of the nth frame in the i-th video. i,n With the feature representation f of the j-th frame i,j Interaction relationship r(f) i,n ,f i,j ):

[0017] r(f i,n ,f i,j )=P×tanh(W1f i,n +W2f i,j +b) (2)

[0018] In equation (2), P, W1, W2 are three parameter matrices to be learned, b is the bias vector, and tanh represents the activation function.

[0019] Step 4.2: The local attention enhancement layer uses equation (3) to calculate the feature representation f of the nth frame in the i-th video. i,n Local attention-enhanced video frame features Thus, the feature representation of the i-th video with local attention enhancement is obtained.

[0020]

[0021] In equation (3), f represents the feature representation of the j-th frame. i,j The feature representation f of the nth frame of the i-th video i,n The relationship weights are denoted by ·, which represents element-wise multiplication of vectors, and we have:

[0022]

[0023] Step 4.3: The fully connected MLP network uses equation (5) to calculate the feature representation f of the nth frame of the i-th video. i,n Predicted score

[0024]

[0025] In equation (5), GeLU represents the activation function; + represents the residual connection;

[0026] Step 4.4: Construct the bisection cross-entropy loss L using equation (7). vsum :

[0027]

[0028] In equation (7), B represents the number of videos in the video subtitle dataset D;

[0029] In the first training phase, the video summarizer is trained using backpropagation and gradient descent based on the video caption dataset D, with a binary cross-entropy loss L. vsum Training stops when the minimum value is reached, thus obtaining a well-trained video summarizer model;

[0030] Step 5: Represent the frame features of the i-th video as F i ={f i,1, f i,2 ,...,f i,n ,..,f i,N The data is input into the trained video summarizer model, and the feature representations of the top K frames with the highest predicted scores are selected to form the sub-optimal video frame set. in, Let K represent the optimal feature representation of the k-th frame of the i-th video; K represents the number of optimal video frames selected.

[0031] Step 6: Construct a decoder consisting of a lightweight Long Short-Term Memory (LSTM) network and train it.

[0032] Step 6.1: When t=1, the optimal video frame set corresponding to the i-th video. The input is processed by the decoder to obtain the predicted word of the m-th subtitle phrase corresponding to the i-th video at time step t.

[0033] When t = 2, 3, ..., W, then the control factor ζ at step t is randomly initialized. t ,like Then the predicted word for the m-th subtitle phrase corresponding to the i-th video output at time step t-1. After processing by the decoder, the predicted word for the m-th subtitle phrase corresponding to the i-th video at time step t is obtained. like Then the t-th word y in the m-th subtitle statement corresponding to the i-th video i,m,t After the t-th word is processed by the decoder, the predicted word for the m-th subtitle phrase corresponding to the i-th video at the t-th time step is obtained.

[0034] Step 6.2: Construct the cross-entropy loss L using equation (8). XE :

[0035]

[0036] In equation (10), p θ (y i,m,t ) represents the decoder's processing of the t-th word y in the m-th subtitle statement corresponding to the i-th video at step t. i,m,t The output is the predicted probability, where θ represents the learning parameters;

[0037] Step 6.3: During the second training phase, based on the English subtitle statement Y... i The decoder is trained using backpropagation and gradient descent, and Y is made... i Training stops when the minimum value is reached, thus obtaining a trained decoder model, which is used to output subtitles for the optimal video frame output by the trained video summarizer model.

[0038] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the multimodal video summarization extraction method, and the processor is configured to execute the program stored in the memory.

[0039] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the multimodal video summarization extraction method.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] 1. This invention utilizes a dual video summarization framework based on video coding, namely the dual coupling of the summarizer and the decoder, to quickly and efficiently compress video content while ensuring the accuracy of semantic information. This improves the efficiency of users browsing short videos and enables the display of short videos on short video websites.

[0042] 2. This invention uses visual and textual modes to summarize video content. It extracts the visual representations of keyframes and then summarizes them based on frame-level scores. A cross-modal video summarization model is then used to select the most meaningful and semantically consistent frames. This compresses the video content without reducing the quality of the video description, eliminates redundant frames, and improves the efficiency of video representation.

[0043] 3. This invention uses a lightweight LSTM decoder to generate descriptions, which can convey the same semantic information without a large number of keyframes, thus bringing beneficial application value to the fields of video encoding and video text data processing. Attached Figure Description

[0044] Figure 1 This is a framework diagram of the multimodal video summarization model of the present invention;

[0045] Figure 2 This is a block diagram of the digester module of the present invention;

[0046] Figure 3 This is a diagram of the subtitle generator module of the present invention;

[0047] Figure 4 This is a flowchart of the training process for the multimodal video summarization model of the present invention. Detailed Implementation

[0048] In this embodiment, a multimodal video summarization extraction method based on video subtitles is described, such as... Figure 1 and Figure 4 As shown, the procedure is as follows:

[0049] Step 1: Obtain the frame feature representation of the video:

[0050] For a video subtitle dataset D = {V, Y}, where V represents the video set and Y represents the set of English subtitles for each video in the video set V;

[0051] A visual encoder using the CLIP model processes any i-th video in the video set V to obtain the frame feature representation F of the i-th video. i ={f i,1, f i,2 ,...,f i,n ,..,f i,N}; where f i,n Let N represent the feature representation of the nth frame in the i-th video, and let N represent the total number of frames in video i. In this embodiment, N = 12.

[0052] Step 2: Obtain the feature representation of the subtitles:

[0053] The text encoder using the CLIP model pairs the English subtitle statement Y corresponding to the i-th video. i ={y i,1,1 ,...,y i,1,W ;y i,m,1 ,y i,m,2 ,...,y i,m,t,..., y i,m,W ;y i,M,1 ,...,y i,M,W The process is performed to obtain the subtitle text vector T corresponding to video i. i ={t i,1 ,t i,2 ,...,t i,m ,..,t i,M}, where y i,m,t This represents the t-th word in the m-th subtitle statement corresponding to the i-th video, where t i,mThis represents the m-th subtitle vector in the corresponding English subtitle statement in the i-th video. In this embodiment, M = 20 and W = 30.

[0054] Step 3, as follows Figure 2 As shown, the feature representation f of the nth frame in the i-th video is obtained using equation (1). i,n With subtitle text vector T i Average similarity s(f i,k ,T i ), and use it as the feature representation of the nth frame of video i. i,n Automated scoring

[0055]

[0056] In equation (1), tr represents the vector transpose;

[0057] Step 4: Construct a video summarizer, including: a self-attention mechanism layer, a local attention enhancement layer, and a fully connected network MLP, and train it;

[0058] Step 4.1: The self-attention mechanism layer uses equation (2) to calculate the feature representation f of the nth frame in the i-th video. i,n With the feature representation f of the j-th frame i,j Interaction relationship r(f) i,n ,f i,j ):

[0059] r(f i,n ,f i,j )=P×tanh(W1f i,n +W2f i,j +b) (2)

[0060] In equation (2), P, W1, W2 are three parameter matrices to be learned, b is the bias vector, and tanh represents the activation function.

[0061] Step 4.2: The local attention enhancement layer uses equation (3) to calculate the feature representation f of the nth frame in the i-th video. i,n Local attention-enhanced video frame features Thus, the feature representation of the i-th video with local attention enhancement is obtained.

[0062]

[0063] In equation (3), f represents the feature representation of the j-th frame of the i-th video. i,j With the feature representation f of the nth frame i,n The relationship weights are denoted by ·, which represents element-wise multiplication of vectors, and we have:

[0064]

[0065] Step 4.3: The fully connected MLP network uses equation (5) to calculate the feature representation f of the nth frame of the i-th video. i,n Predicted score

[0066]

[0067] In equation (5), GeLU represents the activation function; + represents the residual connection;

[0068] Step 4.4: In the first training phase, the video summarizer is trained based on the video caption dataset D using backpropagation and gradient descent, and the binary cross-entropy loss L is minimized as shown in equation (7). vsum To optimize the video summarizer, a well-trained video summarizer model is obtained:

[0069]

[0070] In equation (7), B represents the number of videos in the video caption dataset D.

[0071] In this embodiment, the maximum number of iterations, epoch_number, is set. 1 The value is 10. The gradient descent method uses the Adam optimization algorithm with a learning rate and exponential decay rate. When the number of iterations reaches epoch_number... 1 When training stops, the objective function loss L is reduced. vsum To reach the minimum;

[0072] Step 5: Represent the frame features of the i-th video as F i ={f i,1 ,f i,2 ,...,f i,n ,..,f i,N The data is input into the trained video summarizer model, and the feature representations of the top K frames with the highest predicted scores are selected to form the sub-optimal video frame set. in, Let K represent the optimal feature representation of the k-th frame of the i-th video; K represents the number of optimal video frames selected.

[0073] Step 6: Construct and train a decoder consisting of a lightweight Long Short-Term Memory (LSTM) network, such as... Figure 3As shown, a lightweight LSTM decoder is used to generate the description. Video frames are sparsified at a certain sampling rate to generate video ViT features. These features are used as input to the model's summarizer, which can determine the information content of the input video frames and provide a specific quantitative evaluation. Then, based on the evaluation, the set of frame features with the highest information content is selected and fed into the LSTM decoder to generate the language description.

[0074] Step 6: Construct a decoder consisting of a lightweight Long Short-Term Memory (LSTM) network and train it.

[0075] Step 6.1: When t=1, the optimal video frame set corresponding to the i-th video. The input is processed by the decoder to obtain the predicted word of the m-th subtitle phrase corresponding to the i-th video at time step t.

[0076] When t = 2, 3, ..., W, then the control factor ζ at step t is randomly initialized. t ,like Then the predicted word for the m-th subtitle phrase corresponding to the i-th video output at time step t-1. After processing by the decoder, the predicted word for the m-th subtitle phrase corresponding to the i-th video at time step t is obtained. like Then the t-th word y in the m-th subtitle statement corresponding to the i-th video i,m,t After the t-th word is processed by the decoder, the predicted word for the m-th subtitle phrase corresponding to the i-th video at the t-th time step is obtained.

[0077] Step 6.2: Construct the cross-entropy loss L using equation (8). XE :

[0078]

[0079] In equation (10), p θ (y i,m,t ) represents the decoder's processing of the t-th word y in the m-th subtitle statement corresponding to the i-th video at step t. i,m,t The output is the predicted probability, where θ represents the learning parameters;

[0080] Step 6.3: In the second training phase, based on the English subtitle statement Y... i The decoder is trained using backpropagation and gradient descent, and L is calculated. XE To update network parameters, set the maximum number of iterations, epoch_number. 2The value is 30. In this step, the gradient descent method uses the Adam optimization algorithm with a learning rate and exponential decay rate. When the number of iterations reaches epoch_number... 2 When the time is right, training stops, thus obtaining a trained decoder, which is used to output subtitles for the best video frame output by the trained video summarizer model.

[0081] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0082] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0083] In summary, this method addresses the rise of short videos by outputting a set of keyframes and their corresponding subtitles. The keyframe set, using a relatively small number of video frames, visually reflects the overall content of the video, while the matching subtitles summarize the video footage in text form. This approach reflects the video content from both visual and textual perspectives. The method uses a small number of model parameters and has limited requirements for storage and computing resources, enabling effective deployment and application.

Claims

1. A method for extracting multimodal video summaries based on video subtitles, characterized in that, The procedure is as follows: Step 1: Obtain the frame feature representation of the video: For a video subtitle dataset D = {V, Y}, where V represents the video set and Y represents the set of English subtitles for each video in the video set V; A visual encoder using the CLIP model processes any i-th video in the video set V to obtain the frame feature representation F of the i-th video. i ={f i,1, f i,2 ,...,f i,n ,..,f i,N }; where f i,n Let N represent the feature representation of the nth frame in the i-th video, and N represent the total number of frames in video i. Step 2: Obtain the feature representation of the subtitles: The text encoder using the CLIP model pairs the English subtitle statement Y corresponding to the i-th video. i ={y i,1,1 ,...,y i,1,W ;y i,m,1 ,y i,m,2 ,...,y i,m,t ,...,y i,m,W ;y i,M,1 ,...,y i,M,W The process is performed to obtain the English subtitle text vector T corresponding to video i. i ={t i,1 ,t i,2 ,...,t i,m ,..,t i,M }, where y i,m,t This represents the t-th word in the m-th subtitle statement corresponding to the i-th video, where t i,m This represents the m-th subtitle vector in the corresponding English subtitle text of the i-th video; W represents the total number of words; Step 3: Use equation (1) to obtain the feature representation f of the nth frame in the i-th video. i,n With subtitle text vector T i Average similarity s(f i,k ,T i ), and use it as the feature representation of the nth frame of video i. i,n Automated scoring In equation (1), tr represents the vector transpose; Step 4: Construct a video summarizer, including: a self-attention mechanism layer, a local attention enhancement layer, and a fully connected network MLP, and train it; Step 4.1: The self-attention mechanism layer uses equation (2) to calculate the feature representation f of the nth frame in the i-th video. i,n With the feature representation f of the j-th frame i,j Interaction relationship r(f) i,n ,f i,j ): r(f i,n ,f i,j )=P×tanh(W1f i,n +W2f i,j +b) (2) In equation (2), P, W1, W2 are three parameter matrices to be learned, b is the bias vector, and tanh represents the activation function. Step 4.2: The local attention enhancement layer uses equation (3) to calculate the feature representation f of the nth frame in the i-th video. i,n Local attention-enhanced video frame features Thus, the feature representation of the i-th video with local attention enhancement is obtained. In equation (3), f represents the feature representation of the j-th frame. i,j The feature representation f of the nth frame of the i-th video i,n The relationship weights are denoted by ·, which represents element-wise multiplication of vectors, and we have: Step 4.3: The fully connected MLP network uses equation (5) to calculate the feature representation f of the nth frame of the i-th video. i,n Predicted score In equation (5), GeLU represents the activation function; + represents the residual connection; Step 4.4: Construct the bisection cross-entropy loss L using equation (7). vsum : In equation (7), B represents the number of videos in the video subtitle dataset D; In the first training phase, the video summarizer is trained using backpropagation and gradient descent based on the video caption dataset D, with a binary cross-entropy loss L. vsum Training stops when the minimum value is reached, thus obtaining a well-trained video summarizer model; Step 5: Represent the frame features of the i-th video as F i ={f i,1 ,f i,2 ,...,f i,n ,..,f i,N The data is input into the trained video summarizer model, and the feature representations of the top K frames with the highest predicted scores are selected to form the sub-optimal video frame set. in, Let K represent the optimal feature representation of the k-th frame of the i-th video; K represents the number of optimal video frames selected. Step 6: Construct a decoder consisting of a lightweight Long Short-Term Memory (LSTM) network and train it. Step 6.1: When t=1, the optimal video frame set corresponding to the i-th video. The input is processed by the decoder to obtain the predicted word of the m-th subtitle phrase corresponding to the i-th video at time step t. When t = 2, 3, ..., W, then the control factor ζ at step t is randomly initialized. t ,like Then the predicted word for the m-th subtitle phrase corresponding to the i-th video output at time step t-1. After processing by the decoder, the predicted word for the m-th subtitle phrase corresponding to the i-th video at time step t is obtained. like Then the t-th word y in the m-th subtitle statement corresponding to the i-th video i,m,t After the t-th word is processed by the decoder, the predicted word for the m-th subtitle phrase corresponding to the i-th video at the t-th time step is obtained. Step 6.2: Construct the cross-entropy loss L using equation (8). XE : In equation (10), p θ (y i,m,t ) represents the decoder's processing of the t-th word y in the m-th subtitle statement corresponding to the i-th video at step t. i,m,t The output is the predicted probability, where θ represents the learning parameters; Step 6.3: During the second training phase, based on the English subtitle statement Y... i The decoder is trained using backpropagation and gradient descent, and Y is made... i Training stops when the minimum value is reached, thus obtaining a trained decoder model, which is used to output subtitles for the optimal video frame output by the trained video summarizer model.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the multimodal video summarization extraction method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the multimodal video summarization extraction method of claim 1.

Images