A method and system for automatically generating subtitles for short videos

By using the Transformer model and the DAPs model, combined with 3D convolutional network for video feature extraction and event detection, the problem of short video subtitles generation without audio background is solved, and efficient and accurate subtitle generation is achieved, satisfying the user's shooting and commentary need.

CN113159034BActive Publication Date: 2024-11-15HANGZHOU DIANZI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110442856.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-04-23
Publication Date
2024-11-15
Estimated Expiration
2041-04-23

AI Technical Summary

Technical Problem

The prior art is difficult to automatically generate short video subtitles without audio backgrounds, which leads to the embarrassment of ordinary users talking to themselves when shooting and commenting, and the post-subtitle generation is time-consuming and labor-intensive, and there is a lack of effective technical solutions.

Method used

The end-to-end model based on Transformer is adopted, combined with a 3D convolutional network for video feature extraction, the DAPs model is used for event detection, and the subtitles that fit the video picture are generated through the Transformer model.

Benefits of technology

Automatic subtitle generation for short videos is realized, the matching degree between subtitles and video events is improved, the complexity and time-consuming of the generation process is reduced, and the accuracy and efficiency of subtitle generation is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113159034B_ABST
    Figure CN113159034B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for automatically generating subtitles for short videos. Video feature extraction is performed through a 3D convolutional network to obtain a video feature sequence, and the DAPs model is used to perform event detection on the received video feature sequence to obtain predicted event segments and score each segment. Each event segment obtained is processed separately, and first a visual embedding operation is performed on the event segment, and then the event segment after the visual embedding operation is input into a Transformer model to obtain a predicted text. The present invention integrates cross-modal technology, and the early processing includes event detection, which narrows the scope of subsequent text generation work and increases the matching degree between the generated text and the event. The text generation part gives full play to the excellent performance of Transformer in feature encoding and decoding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and relates to text generation technology in the field of natural language processing. The text generation technology is applied to information extraction tasks to generate an understanding of the information. Specifically, it extracts video information, generates a simple textual summary, and finally presents the summary in the form of subtitles. Background Art

[0002] Short videos refer to high-frequency video content played on various new media, suitable for viewing on the go and in a short leisure state, and generally less than 5 minutes long. Since 2016, short videos have quickly become popular due to their low creation threshold, strong social attributes, and fragmented entertainment. The short video industry has developed rapidly in China, and now it has become increasingly mature. Currently, active short video platforms include Weibo, Douyin, Kuaishou, Xiaohongshu, etc. Short videos have become a carrier for bloggers to record and share their lives, a new way for audiences to entertain and promote, and can even promote brands and seek benefits for original authors.

[0003] The creation of short videos consists of content planning, video shooting, post-processing and other links. Among them, subtitle generation, as a task in post-processing, is generally added manually by the author. Manually generating subtitles is time-consuming and energy-consuming. At present, some subtitle generation software for videos has been developed and applied. Based on the voice in the video, text subtitles are exported through voice recognition technology. On this basis, it also has powerful functions such as manual modification and translation, and is highly practical. However, the above software mainly relies on the audio information in the video. Without clearly pronounced audio information, text cannot be extracted. In real life, ordinary users want to record their lives through short videos. If they shoot and explain at the same time, they will inevitably be embarrassed to talk to themselves. If they record it later, it will be time-consuming and laborious. According to the survey, there are pain points and technical gaps in the automatic generation of subtitles for short videos without audio background.

[0004] Text generation technology has a wide range of application scenarios, and can be applied to tasks such as information extraction, question-answering systems, and text creation. Through question-answering systems, chatbots were born, and through text creation, robots can write lyrics and compose music. At the same time, cross-modal text generation has also been a research hotspot in recent years. Cross-modality requires the use of image, audio, and language processing technologies. Common applications include "speaking with pictures" and automatically generating subtitles for teaching videos.

[0005] The technology of automatic subtitle generation is cross-modal, relying on the integration of image vision technology and natural language processing technology. In terms of the overall framework, in 2015, "Sequence to Sequence-Video to Text" proposed a model based on the LSTM framework, which processes the original image and optical flow image separately, uses a 2D convolutional neural network to extract the image features of each frame in the video as the input of the LSTM encoding layer, and the output of the LSTM decoding layer is the prediction result. The probability of the occurrence of words is predicted one by one, and finally the probability of the original image generation and the probability of the optical flow image generation are weighted and summed; in 2015, "Video Description Generation Incorporating Spatio-Temporal Features and a Soft-Attention Mechanism" borrowed the attention mechanism in image description, first extracting features from each frame of the video. In the article, the GoogleNet network is used to extract 2D features, and the 3D convolutional network is used to extract 3D features. The attention mechanism is used every time a word is predicted, the weight of each feature is calculated, and the image features are weighted and summed according to the weight; in 2016, "Describing Videos using Multi-modal Fusion" proposes to integrate image features, video features, environmental sound features, voice features and category features as the feature representation of the video. The so-called fusion is to take the weighted average of each feature and then generate a video description through the LSTM model.

[0006] The innovations of video description technology can come from three aspects: first, optimizing the extraction of video features; second, optimizing the event detection model; and third, optimizing the text generation model.

[0007] "Learning Spatiotemporal Features with 3D Convolutional Networks" proposes 3D convolution, which is suitable for extracting spatiotemporal features and can better focus on information in the time dimension.

[0008] The purpose of event detection is to divide a long video into multiple segments based on semantics, so that each segment contains an event. "Deep Action Proposals for Action Understanding" proposed the event detection model DAPs. The model before DAPs needed to set different sliding windows to scan the entire video multiple times, and then use the maximum likelihood estimation method to find the most suitable event segment, which ran very slowly. DAPs only uses one sliding window to obtain event segments of different scales, greatly improving the running speed and accuracy. "Attention is all you need" proposed the Transformer model, which is suitable for natural language processing tasks such as machine translation, text generation, and text summarization. The entire network is completely composed of the Attention mechanism, which fully pays attention to the contribution of contextual information. Summary of the invention

[0009] The application scenario of the invention is to automatically generate subtitles for short videos such as Weibo, Douyin, and Kuaishou. The model used by the invention takes video features as input and description text as output, based on the end-to-end model of Transfomer. The main tasks are: first, extract video features from video clips; second, segment the video to cut multiple clips according to events; third, generate a summary describing the video event.

[0010] The technical problems to be solved by the present invention are: first, designing a video feature extraction model to optimize the extraction of video features; second, designing a video segmentation model to improve the accuracy of video segmentation; and third, designing a text generation model to make the generated text fit the picture shown in the video.

[0011] In view of the deficiencies in the prior art, the present invention provides a method and system for automatically generating subtitles for a short video.

[0012] A system for automatically generating subtitles for short videos, comprising a video feature extraction module, an event detection module and a text generation module;

[0013] The video feature extraction module extracts video features through a 3D convolutional network to obtain a video feature sequence, and sends the obtained video feature sequence to the event detection module;

[0014] The event detection module uses the DAPs model to perform event detection on the received video feature sequence, inputs the video feature sequence into the LSTM network in the DAPs model, concatenates the features, and uses the hidden layer vector of the LSTM network as the time feature, and then uses a sliding window to scan the entire feature sequence to obtain the predicted event segment and score each segment.

[0015] The text generation module uses the Transformer model to generate text, and processes each event fragment obtained by the event detection module separately. First, a visual embedding operation is performed on the event fragment, and then the event fragment after the visual embedding operation is input into the Transformer model to obtain the predicted text.

[0016] A method for automatically generating subtitles for a short video, the steps are as follows:

[0017] Step (1), constructing a data set;

[0018] We constructed a dataset and randomly sampled videos with subtitles from major domestic short video platforms. We preprocessed the videos uniformly, manually checked the subtitles of each video, and corrected the descriptions that did not match the facts to ensure that the subtitles could fully and concisely summarize the scenes or events shown in the video. The processed subtitles were used as video labels.

[0019] Step (2), video feature extraction;

[0020] The 3D convolutional network in the video feature extraction module is used to extract video features to obtain a video feature sequence. The reason is that the 3D convolutional network is more suitable for learning spatiotemporal features than the 2D convolutional network. The 3D convolutional network can better model temporal information through 3D convolution kernels and 3D pooling operations, and extract video features through convolutional layers, pooling layers, and fully connected layers.

[0021] Step (3), event detection;

[0022] Use the DAPs model in the event detection module to detect events. Based on step 2, input the acquired video features into the LSTM network in the DAPs model, concatenate the features, and use the hidden layer vector of the LSTM network as the time feature. Then use the sliding window to scan the entire feature sequence to obtain the predicted event segment and score each segment. Use each segment score and segment matching accuracy as the loss function to train the DAPs model.

[0023] Step (4), text generation;

[0024] The advantage of using the Transformer model in the text generation module for text generation is that it completely relies on the self-attention mechanism to understand the global dependency between input and output, helps parallelize calculations, and can directly calculate the correlation between each word without passing through the hidden layer. Each event segment obtained in step (3) is processed separately. First, the event segment is visually embedded, and then the event segment after the visual embedding operation is input into the Transformer model to obtain the predicted text.

[0025] Step (5), training the network model of steps (2) to (4) through the data set, and completing the automatic generation of short video subtitles through the trained network model.

[0026] The beneficial effects of the present invention are as follows:

[0027] 1. Crawling short videos and subtitles from Weibo, Douyin, and Kuaishou to contribute data sets. 2. Integrating cross-modal technology, the early processing includes event detection, which narrows the scope of subsequent text generation work and increases the matching degree between the generated text and the event. The text generation part takes advantage of the Transformer's outstanding performance in feature encoding and decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of 3D convolution operation;

[0029] Figure 2 It is a schematic diagram of the C3D network structure;

[0030] Figure 3 Generate flowcharts for text;

[0031] Figure 4 Schematic diagram of the Transformer network structure;

[0032] Figure 5 Schematic diagram of the Transformer macro network structure. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0034] The present invention provides a method and system for automatically generating subtitles for a short video.

[0035] A method for automatically generating subtitles for a short video, comprising the following steps:

[0036] Step (1), constructing a data set;

[0037] We constructed a dataset and randomly sampled videos with subtitles from major domestic short video platforms. We preprocessed the videos uniformly, manually checked the subtitles of each video, and corrected the descriptions that did not match the facts to ensure that the subtitles could fully and concisely summarize the scenes or events shown in the video. The processed subtitles were used as video labels.

[0038] Step (2), video feature extraction.

[0039] The 3D convolutional network used is the C3D network. C3D is a deep 3D convolutional network used to process videos (a collection of images with time sequence). Its operations are as follows: Figure 1As shown in the figure, k is the length and width of the 3D convolution kernel, d is the depth of the 3D convolution kernel, W and H are the length and width of each frame in the video, and L is the number of frames in the video. In the spatial dimension, like the 2D convolution operation, the convolution kernel traverses the image set according to the set step size, and captures the video features through the 3D convolution operation. The complete network framework of C3D is shown in the figure. Figure 2 As shown in the figure, the C3D network includes 8 convolutional layers, 5 maximum pooling layers, 2 fully connected layers, and a nonlinear output layer. The size of the 3D convolution kernel (Conv1a-Conv5b) is 3×3×3, and the stride is 1 in both space and time. The 5 maximum pooling layers are pool1-pool5, the pooling kernel of pool1 is 1×2×2, and the other pooling kernels are 2×2×2. The 2 fully connected layers (fc6-fc7) have 4096 output units. The video is processed by the C3D network to obtain the video feature sequence.

[0040] Step (3), event detection:

[0041] Due to the different lengths of event segments, the model before DAPs needs to set different sliding windows to scan the entire video multiple times, and then use the maximum likelihood estimation method to find the most suitable event segment, which runs very slowly. DAPs only uses one sliding window to obtain event segments of different scales, which greatly improves the running speed. The video feature sequence obtained in step (2) is input into the LSTM network of the DAPs model to connect these features in series. The hidden layer output of the LSTM is used as the time feature sequence. Then, the sliding window is used to scan the entire feature sequence through the Anchor mechanism to obtain the predicted event segment and score each segment. The center of the sliding window is the mapping point on the original video sequence called the Anchor. Proposals of different scales are generated with the Anchor as the center. For each proposal, a score is given based on whether it contains an event, and the matching accuracy of the proposal and the actual event segment is calculated. The loss function is composed of the score of whether the predicted event exists and the matching accuracy. The former requires that the probability of the proposal containing the event is as high as possible, and the latter requires that the predicted segment fits the interval of the actual event segment as much as possible.

[0042] Finally, the result of event detection is obtained: <start time of event 1, end time of event 1>, …, <start time of event n, end time of event n>.

[0043] Step (4), text generation:

[0044] Using the results of event detection, a video clip of each event is obtained, and a video description is performed on each event clip separately in step (4). Figure 3As shown in the figure, the text generation model adopts a Transformer-based text generation model. First, visual embedding is performed, and the video sequence of the event clip is input into the C3D network to extract video features, and then input into the LSTM network to share the timing information, and the hidden layer output of the LSTM is used as the input of the Transformer encoder.

[0045] The Transformer model structure is as follows Figure 4 As shown in Figure 1, it consists of two parts: an encoder and a decoder. The input of the encoder is a video feature sequence, the input of the decoder is an encoded feature sequence, and the output of the decoder is the predicted text. The encoder consists of 6 encoding blocks, and the decoder consists of 6 decoding blocks, as shown in Figure 1. Figure 5 shown.

[0046] Specifically, an encoding block / decoding block contains a multi-head self-attention mechanism layer, a residual normalization layer, and a feedforward neural network layer. The encoding block and the decoding block have the same network structure, but their input and output contents are different.

[0047] The structure and function of each layer are as follows:

[0048] Multi-head self-attention mechanism layer, first the formula of a single-layer self-attention mechanism is:

[0049]

[0050] Among them, Q is the query vector, K is the key vector, and V is the value vector. In the encoding block, Q, K, and V all come from the input of the previous layer of the encoding block or decoding block. The multi-head self-attention mechanism performs multiple linear transformations on the input of the encoding block or decoding block, and then calculates the Attention value respectively, and then concatenates and linearly transforms the results. The formula is:

[0051]

[0052] MultiHead(Q,K,V)=Concat(head1,…,head h )W o

[0053] in, is the linear transformation weight for Q, Similarly, head i is the i-th attention score, concat is the column-by-column concatenation operation, W o is the linear transformation weight.

[0054] Feed Forward neural network layer: It consists of a Relu activation layer and a fully connected layer. Its purpose is to adjust the output dimension. The formula is:

[0055] FFN(x)=nax(0,xW1+b1)W2+b2

[0056] Among them, W1 is the activation layer weight, b1 is the activation layer bias, W2 is the fully connected layer weight, b2 is the fully connected layer bias, and x is the input vector.

[0057] Residual normalization layer (Add&Norm): It consists of a residual network and normalization. The residual connection solves the problem of multi-layer network training, allowing the network to focus only on the current difference. The normalization operation can speed up the network convergence. The formula is:

[0058] output = LayerNorm(x + F(x))

[0059] Among them, x is the input vector, F(x) is the previous layer operation of the residual normalization layer, the multi-head self-attention mechanism or the feedforward neural network.

[0060] Step (5), training the network model of steps (2) to (4) through the data set, and completing the automatic generation of short video subtitles through the trained network model.

[0061] A system for automatically generating subtitles for short videos, comprising a video feature extraction module, an event detection module and a text generation module;

[0062] The video feature extraction module extracts video features through a 3D convolutional network to obtain a video feature sequence, and sends the obtained video feature sequence to the event detection module;

[0063] The event detection module uses the DAPs model to perform event detection on the received video feature sequence, inputs the video feature sequence into the LSTM network in the DAPs model, concatenates the features, and uses the hidden layer vector of the LSTM network as the time feature, and then uses a sliding window to scan the entire feature sequence to obtain the predicted event segment and score each segment.

[0064] The text generation module uses the Transformer model to generate text, and processes each event fragment obtained by the event detection module separately. First, a visual embedding operation is performed on the event fragment, and then the event fragment after the visual embedding operation is input into the Transformer model to obtain the predicted text.

Claims

1. A method for automatically generating subtitles for short videos, characterized in that: Here are the steps: Step (1), constructing a data set; Construct a data set, randomly sample videos with subtitles from major domestic short video platforms, pre-process the videos, manually check the subtitles of each video, correct the descriptions that do not match the facts, and confirm that the subtitles can fully and concisely summarize the scenes or events shown in the video. The processed subtitles are used as video labels; Step (2), video feature extraction; Use the 3D convolutional network in the video feature extraction module to extract video features and obtain a video feature sequence; Step (3), event detection; Use the DAPs model in the event detection module to detect events. Based on step 2, input the acquired video features into the LSTM network in the DAPs model, concatenate the features, and use the hidden layer vector of the LSTM network as the time feature. Then use the sliding window to scan the entire feature sequence to obtain the predicted event segment and score each segment. Use the score of each segment and the segment matching accuracy as the loss function to train the DAPs model. Step (4), text generation; Use the Transformer model in the text generation module to generate text. Process each event segment obtained in step (3) separately. First, perform a visual embedding operation on the event segment. Then, input the event segment after the visual embedding operation into the Transformer model to obtain the predicted text. Step (5), training the network model of steps (2) to (4) using the data set, and completing the automatic generation of short video subtitles using the trained network model; The specific method of step (2) is as follows: The 3D convolutional network used is the C3D network. C3D is a deep 3D convolutional network used to process videos. The C3D network includes 8 convolutional layers, 5 maximum pooling layers, 2 fully connected layers, and a nonlinear output layer. The 3D convolution kernel size is 3×3×3, and the step size is 1 in both space and time. The 5 maximum pooling layers are pool1-pool5, the pooling kernel of pool1 is 1×2×2, and the other pooling kernels are 2×2×2. The 2 fully connected layers have 4096 output units. The video is processed through the C3D network to obtain a video feature sequence. The specific method of step (3) is as follows: The video feature sequence obtained in step (2) is input into the LSTM network of the DAPs model to connect these features in series. The hidden layer output of the LSTM is used as the time feature sequence. Then, the entire feature sequence is scanned using a sliding window through the Anchor mechanism to obtain the predicted event segment and score each segment. The mapping point of the center of the sliding window on the original video sequence is called the Anchor. Proposals of different scales are generated with the Anchor as the center. For each proposal, a score is given based on whether it contains an event, and the matching accuracy between the proposal and the actual event segment is calculated. The loss function is composed of the score of whether the predicted event exists and the matching accuracy. The former requires that the probability of the proposal containing the event is as high as possible, and the latter requires that the predicted segment fits the interval of the actual event segment as much as possible. Finally, the result of event detection is obtained: <start time of event 1, end time of event 1>, ..., <start time of event n, end time of event n>; The specific method of step (4) is as follows: The text generation model uses a Transformer-based text generation model. First, visual embedding is performed, and the video sequence of the event clip is input into the C3D network to extract video features, and then input into the LSTM network to share the timing information, and the hidden layer output of the LSTM is used as the input of the Transformer encoder. The Transformer model consists of two parts: an encoder and a decoder. The input of the encoder is a video feature sequence, the input of the decoder is an encoded feature sequence, and the output of the decoder is the predicted text. The encoder consists of 6 encoding blocks, and the decoder consists of 6 decoding blocks. Specifically, an encoding block / decoding block contains a multi-head self-attention mechanism layer, a residual normalization layer, and a feedforward neural network layer; the encoding block and the decoding block have the same network structure, but their input and output contents are different; The structure and function of each layer are as follows: Multi-head self-attention mechanism layer, first the formula of a single-layer self-attention mechanism is: Among them, Q is the query vector, K is the key vector, and V is the value vector. In the encoding block, Q, K, and V all come from the input of the previous layer of the encoding block or the decoding block; the multi-head self-attention mechanism performs multiple linear transformations on the input of the encoding block or the decoding block, and then calculates the Attention value respectively, and then concatenates and linearly transforms the results. The formula is: MultiHead(Q,K,V)=Concat(head1,…,head h )W o in, is the linear transformation weight for Q, Similarly, head i is the i-th attention score, concat is the column-by-column concatenation operation, W o is the linear transformation weight; Feedforward neural network layer: It consists of a Relu activation layer and a fully connected layer. The purpose is to adjust the output dimension. The formula is: FFN(x)=max(0,xW1+b1)W2+b2 Among them, W1 is the activation layer weight, b1 is the activation layer bias, W2 is the fully connected layer weight, b2 is the fully connected layer bias, and x is the input vector; Residual normalization layer: It is composed of residual network and normalization. Residual connection solves the problem of multi-layer network training, allowing the network to focus only on the current difference. Normalization operation can speed up network convergence. The formula is: output = LayerNorm(x + F(x)) Among them, x is the input vector, F(x) is the previous layer operation of the residual normalization layer, the multi-head self-attention mechanism or the feedforward neural network.

2. A system for automatically generating subtitles for short videos as claimed in claim 1, characterized in that: It includes video feature extraction module, event detection module and text generation module; The video feature extraction module extracts video features through a 3D convolutional network to obtain a video feature sequence, and sends the obtained video feature sequence to the event detection module; The event detection module uses the DAPs model to perform event detection on the received video feature sequence, inputs the video feature sequence into the LSTM network in the DAPs model, concatenates the features, and uses the hidden layer vector of the LSTM network as the time feature, and then uses a sliding window to scan the entire feature sequence to obtain the predicted event segment and score each segment; The text generation module uses the Transformer model to generate text, and processes each event fragment obtained by the event detection module separately. First, a visual embedding operation is performed on the event fragment, and then the event fragment after the visual embedding operation is input into the Transformer model to obtain the predicted text.