A video summarization method based on transformer and TCN

Abstract

The application discloses a video abstracting method based on a transformer and a TCN, and comprises the following steps: performing feature extraction by using a pre-trained convolutional network; extracting time sequence information by using a time sequence convolutional network; inputting spatial feature sequences and time sequence feature sequences of a video into a local transformer module and a global transformer module respectively, calculating the interframe correlation and the long-term dependence between video sequence nodes, and fusing the features; predicting the importance of each frame; dividing the video into segments, calculating the importance of each segment, selecting segments with high importance, and generating a dynamic abstracting. The above process trains a video abstracting model in a supervised manner by using a labeled daily video abstracting data set. The application emphasizes the extraction and separation of interframe correlation and long-term dependence in the video abstracting task on the basis of time sequence modeling, and improves the understanding ability of the model for the video.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically a video summarization method based on Transformer and TCN. Background Technology

[0002] With the development of high-speed internet and low-cost storage, recording videos has become a common way for people to capture moments in daily life. As more and more videos are stored on devices like phones and computers, people want to quickly understand the main content of each video through concise video summaries, enabling faster browsing. Therefore, automatic video summarization technology is crucial. In recent years, the amount of video data has grown exponentially. For example, on YouTube, users upload a total of 720,000 hours of videos daily, including tutorials, guides, news, educational videos, product reviews, and many other types. However, most viewers are more interested in interesting scenes within a video than the entire video. For instance, with surveillance videos, people are only interested in segments featuring specific events, and not segments without people or specific events. Manually extracting segments of interest from massive amounts of video data requires a significant amount of work; therefore, automatic video summarization technology is essential. The goal of video summarization is to automatically generate concise summaries that contain the information of the entire original video. Video summarization automatically removes redundant information from the original video, retaining only the content of interest to the viewer, allowing them to understand the overall information of the video simply by looking at the summary.

[0003] For video summarization tasks, the summary should meet the following requirements: (1) The generated summary should include the most important shots and scenes in the original video. (2) The length of the generated summary should be much shorter than the length of the original video. (3) There should be significant differences between the generated summaries. (4) Based on the generated summary, most of the content of the original video can be reconstructed. Since the generated summary needs to meet the above conditions, video summarization is a very challenging problem. For video summarization, the importance of each frame cannot be determined solely by the local information of the frame itself, but depends on the inter-frame relationships and the position of each frame in the entire video. In video summarization tasks, inter-frame correlation information is extremely important, and the inter-frame relationships in the video are quite complex. Therefore, establishing inter-frame relationships is a challenging problem. In addition, focusing only on local inter-frame relationships cannot meet the requirements of video summarization tasks. The generation of the summary needs to focus on the content of the entire video, and the relationship between two frames that are far apart is difficult to model.

[0004] dpp-LSTM extracts temporal information through LSTM (Zhang K, Chao WL, Sha F, et al. Video summarization with long short-term memory[C] / / Computer Vision–ECCV 2016:14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14. Springer International Publishing, 2016:766-782.). However, in the LSTM structure, the hidden state of each node is only directly affected by the hidden state of the previous node. In video summarization tasks, the criticality of each frame is determined by the other frames. Therefore, LSTM cannot handle this situation well. At the same time, LSTM cannot focus on long-term dependencies in video sequences and cannot handle long videos. VASNet extracts temporal information through attention (Fajtl J, Sokeh HS, Argyriou V, et al. Summarizing videos with attention [C] / / Computer Vision–ACCV 2018Workshops: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers)

[0005] 14. Springer International Publishing, 2019: 39-54.) Compared to LSTM, attention can directly focus on inter-frame correlation, which is a better modeling method. In addition, the attention mechanism can solve the long-term dependency problem in video. However, the attention mechanism does not reflect the positional information of each frame in the sequence, which is not conducive to summarization. PGL-SUM uses Transformer to extract inter-frame correlation (Apostolidis E, Balaouras G, Mezaris V, et al. Combining global and local attention with positional encoding for videosummarization[C] / / 2021IEEE international symposium on multimedia(ISM).IEEE, 2021: 226-234.) and adds position encoding to the input feature sequence to inject information about the token position into the sequence. However, during the addition process, position encoding may cover the information of the original features. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a video summarization method based on Transformer and TCN. The Transformer extracts inter-frame correlations, and the TCN extracts temporal information, thus eliminating the role of position encoding in injecting positional information. At the same time, this invention separates the extraction process of local inter-frame relationships from the extraction process of global long-term dependencies, reducing the deviation of attention weights during the calculation process.

[0007] The present invention is achieved by at least one of the following technical solutions.

[0008] A video summarization method based on Transformer and TCN includes the following steps:

[0009] Step 1: Input the video frame sequence into the pre-trained convolutional network model for feature extraction to obtain the spatial feature sequence S of each frame;

[0010] Step 2: Use the spatial feature sequence S of the video as the input of the temporal convolutional network, and output the temporal feature sequence R of the video;

[0011] Step 3: Using the spatial feature sequence S and temporal feature sequence R of the video as input to the local Transformer module, calculate the local inter-frame correlation and output the temporal feature sequence F incorporating the local inter-frame correlation. (l) ;

[0012] Step 4: Using the spatial feature sequence S and temporal feature sequence R of the video as input to the global Transformer module, calculate the long-term dependencies between nodes in the global video sequence and output the temporal feature sequence F incorporating the long-term dependencies. (g) ;

[0013] Step 5: Incorporate the temporal feature sequence F with local inter-frame correlation. (l) With time-series feature sequences F incorporating long-term dependencies (g) Feature fusion is performed and the data is input into a fully connected network to predict the importance of each frame;

[0014] Step 6: Train the above network model in a supervised manner using a labeled dataset; divide the original video into segments and input them into the trained model, calculate the importance of each segment, and select the segments with high importance to generate a dynamic summary.

[0015] Furthermore, in step 1, the pre-trained convolutional network model is the GoogleNet network, and features are extracted through the pre-trained GoogleNet network.

[0016] Furthermore, in steps 3 and 4, the linear transformation of the spatial feature sequence is used as the query value and fed into the local Transformer module and the global Transformer module, respectively; the linear transformation of the temporal feature sequence is used as the key and value and fed into the local Transformer module and the global Transformer module, respectively.

[0017]

[0018] Where S represents the spatial feature sequence, R represents the temporal feature sequence, and W... m K W m V W m Q K represents the mapping matrix of the key, value, and query item of the m-th head parameter, respectively. m V m Q m These represent the key, value, and query of the m-th head, respectively.

[0019] Furthermore, in step 3, the local Transformer module only calculates the attention weights between the current frame and its local neighboring frames:

[0020]

[0021] MH (l) =Concat(H1) (l) H2 (l) H M (l) )

[0022] F (l) =MH (l) +S

[0023] in This represents the vector corresponding to the i-th node in the value sequence of the m-th head. This represents the vector corresponding to the t-th node in the query sequence of the m-th head. The vector representing the i-th node in the key sequence of the m-th head, nerb(t) represents the neighboring frame; H M (l) h represents the output of the m-th head in the local Transformer. m，t (l) Represents H M (l) The vector corresponding to the t-th node; MH (l) F represents the result of the multi-head attention mechanism. (l) This represents the output of the local Transformer module, where S represents the spatial feature sequence.

[0024] Furthermore, in step 3, the method for dividing the local and global regions is a sliding window:

[0025] nerb(t)={tl,t-l+1,…,t,…,t+l}

[0026] Where l represents the size of the neighborhood window, and nerb(t) represents the neighborhood of the t-th node.

[0027] Furthermore, in step 4, the global Transformer module only calculates the attention weights between the current frame and other frames outside the local neighborhood:

[0028]

[0029] MH (g) =Concat(H1) (g) H2 (g) H M(g) )

[0030] F (g) =MH (g) +S

[0031] in This represents the vector corresponding to the i-th node in the value sequence of the m-th head. This represents the vector corresponding to the t-th node in the query sequence of the m-th head. The vector representing the i-th node in the key sequence of the m-th head, nerb(t) represents the neighboring frame, and H M (g) h represents the output of the m-th head in the global Transformer. m，t (g) Represents H M (g) The vector corresponding to the t-th node, MH (g) F represents the result of the multi-head attention mechanism. (g) This represents the output of the global Transformer module, where S represents the spatial feature sequence.

[0032] Furthermore, in step 4, the method for dividing the local and global regions is a sliding window:

[0033] nerb(t)={tl,t-l+1,…,t,…,t+l}

[0034] Where l represents the size of the neighborhood window, and nerb(t) represents the neighborhood of the t-th node.

[0035] Furthermore, in step 5, feature fusion is performed by adding features together:

[0036] F = F (g) +F (l)

[0037] Where F (g) F represents the output of the global Transformer module. (l) This represents the output of the local Transformer module, and F represents the result of feature fusion.

[0038] Furthermore, in step 6, the fragments are divided using the KTS algorithm.

[0039] Furthermore, in step 6, the 0-1 knapsack algorithm is used to select segments with high importance to generate a dynamic summary. Compared with existing technologies, the beneficial effects of this invention are:

[0040] This invention proposes a video summarization method based on Transformer and TCN for everyday life videos. The TCN network models temporal information, focusing on the sequential information between nodes in the sequence; the attention module in the Transformer models inter-frame correlations and extracts inter-frame relationships. Furthermore, during the Transformer computation, this invention separates the extraction of global long-term dependencies from the extraction of local inter-frame correlations, reducing the bias of the attention weights obtained during computation and thus better modeling the two types of relationships separately. Therefore, compared to LSTM-based video summarization methods, the proposed method can better model inter-frame correlations and long-term video dependencies, uncovering deeper semantic information in the video. Attached Figure Description

[0041] Figure 1 This is a flowchart of a video summarization method based on Transformer and TCN according to an embodiment of the present invention;

[0042] Figure 2 This is a network model diagram for a video summarization method based on Transformer and TCN.

[0043] Figure 3 This is a graph showing the inter-frame weight relationships within a local neighborhood.

[0044] Figure 4 This is a graph showing the weight relationships between frames outside the local neighborhood. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] like Figure 1 , Figure 2 As shown, this embodiment of a video summarization method based on Transformer and TCN establishes a video summarization model to mine deeper semantic information from videos. Specifically, it includes the following steps:

[0047] S1. Input the video frame sequence into the pre-trained convolutional network model to extract features and obtain the spatial feature sequence of each frame.

[0048] Specifically, given a video sequence, let it be denoted as Where x i This represents the i-th frame of the video, where I represents the total number of video frames.

[0049] As one example, feature extraction is performed using a pre-trained GoogleNet model to obtain a video spatial feature sequence. s i =f g (x i ), where f g Representing the GoogleNet model, s i This represents the i-th node in the spatial feature sequence.

[0050] S2. Using the spatial feature sequence of the video as the input of the temporal convolutional network, the output is the temporal feature sequence of the video;

[0051] Specifically, using video spatial feature sequences As input to a temporal convolutional network, it outputs a sequence of video temporal features. R = f tcn (S), where r i This represents the i-th node in the time-series feature sequence.

[0052] S3. Using the spatial feature sequence S and temporal feature sequence R of the video as input to the local Transformer module, calculate the local inter-frame correlation and output the temporal feature sequence F incorporating the local inter-frame correlation. (l) ;

[0053] Specifically, based on spatial feature sequences Calculate the query value based on the time-series feature sequence. Calculate the key and value:

[0054]

[0055] Among them W m K W m V W m Q K represents the mapping matrix of the key, value, and query item of the m-th head parameter, respectively. m V m Q m These represent the key, value, and query of the m-th head, respectively.

[0056] The neighborhood is divided using a sliding window: nerb(t) = {tl, t-l+1, ..., t, ..., t+l}, where l represents the size of the neighborhood window and nerb(t) represents the neighborhood of the t-th node. The attention between local frames of each head is calculated to obtain the output. In this process, attention weights within the neighborhood are calculated based on the neighborhood partitioning, such as... Figure 3 As shown, the dark part represents the attention weight between two frames that is considered during the attention calculation process, while the light part represents the attention weight between two frames that is ignored during the attention calculation process.

[0057]

[0058] in This represents the vector corresponding to the i-th node in the value sequence of the m-th head. This represents the vector corresponding to the t-th node in the query sequence of the m-th head. The vector representing the i-th node in the key sequence of the m-th head, nerb(t) represents the neighboring frame, and h m，t (l) This represents the vector corresponding to the t-th node in the output sequence of the m-th head in the local Transformer.

[0059] Calculate the output of the local Transformer based on the attention output of each head.

[0060] MH (l) =Concat(H1) (l) H2 (l) H M (l) )

[0061] F (l) =MH (l) +S

[0062] Among them MH (l) F represents the result of the multi-head attention mechanism. (l) H represents the output of the local Transformer module. M (l) This represents the output of the m-th head in the local Transformer.

[0063] S4. Using the spatial feature sequence S and temporal feature sequence R of the video as input to the global Transformer module, calculate the long-term dependencies between nodes in the global video sequence and output the temporal feature sequence F incorporating global inter-frame correlations. (g) ;

[0064] Specifically, based on spatial feature sequences Calculate the query value based on the time-series feature sequence. Calculate the key and value:

[0065]

[0066] Where K m V m Q m These represent the key, value, and query of the m-th head, respectively.

[0067] The neighborhood is divided using a sliding window: nerb(t) = {tl, t-l+1, ..., t, ..., t+l}, where l represents the size of the neighborhood window and nerb(t) represents the neighborhood of the t-th node. The inter-frame attention outside the local scope of each head is calculated to obtain the output. In this process, attention weights outside the neighborhood are calculated based on the neighborhood partitioning, such as... Figure 4 As shown, the dark part represents the attention weight between two frames that is considered during the attention calculation process, while the light part represents the attention weight between two frames that is ignored during the attention calculation process.

[0068]

[0069] in This represents the vector corresponding to the i-th node in the value sequence of the m-th head. This represents the vector corresponding to the t-th node in the query sequence of the m-th head. The vector representing the i-th node in the key sequence of the m-th head, nerb(t) represents the neighboring frame, and h m，t (g) This represents the vector corresponding to the t-th node in the output sequence of the m-th head in the global Transformer.

[0070] Calculate the global Transformer output based on the attention output of each head.

[0071] MH (g) =Concat(H1) (g) H2 (g) H M (g) )

[0072] F (g) =MH (g) +S

[0073] Among them MH (g) F represents the result of the multi-head attention mechanism. (g) This represents the output of the global Transformer module. H M (g) This represents the output of the m-th head in the global Transformer.

[0074] S5. Perform feature fusion between the temporal feature sequence incorporating local inter-frame correlation and the temporal feature sequence incorporating long-term dependency, and input the fusion into the fully connected network to predict the importance of each frame.

[0075] Specifically, feature fusion is performed by adding the local Transformer output to the global Transformer output, where F = F0. (l) +F (g) Output fusion result Input F into a fully connected network to output the importance of each frame:

[0076] D t =FFN(f t )

[0077] p t =sigmoid(W D D t +b D )

[0078] Where FFN represents the feedforward network, f t D represents the fused feature corresponding to the input frame t. t p represents the output of the feedforward network. t W represents the importance of frame t. D With b D This represents the weight and bias parameters of the linear layer.

[0079] S6, Training Model

[0080] Specifically, this invention trains the model in a supervised manner using the labeled datasets SUMME and TVSUM. When frame t is selected from the manually annotated summary, this invention treats the importance of frame t as 1, and otherwise as 0. Based on the annotations from multiple users in the dataset, the average importance of each frame is calculated and used as the ground truth label. The loss function is defined using MSE.

[0081]

[0082] Where g t This represents the groundtruth tag.

[0083] S7. Divide the video into segments, calculate the importance of each segment, select the segments with high importance, and generate a dynamic summary.

[0084] Specifically, based on the trained model, the KTS algorithm is used to segment the original video sequence. As input, segments are divided, and the average importance of each segment is calculated.

[0085]

[0086] Where J represents the total number of segments, K j N represents the j-th segment. j y represents the number of frames contained in the j-th segment. j This represents the importance of the j-th segment.

[0087] As one example, based on the importance of each segment, a video summary with a length not exceeding 15% is selected using a 0-1 knapsack algorithm as the final generated summary.

[0088] To verify the effectiveness of this invention, experiments were conducted on the SUMME and TVSUM datasets, and the F1 score was used for model evaluation. First, the datasets were divided five times according to the video summarization testing criteria. The model was trained and tested in each division, and the average result was taken. The experimental results are shown in Table 1.

[0089] Table 1 Comparison of Experimental Results

[0090]

[0091] It can be observed that, compared with other advanced methods, the results of this invention can achieve good results on both the SUMME and TVSUM datasets. In summary, the proposed method can effectively focus on inter-frame correlation.

[0092] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A video summarization method based on Transformer and TCN, characterized in that, Includes the following steps: Step 1: Input the video frame sequence into the pre-trained convolutional network model for feature extraction to obtain the spatial feature sequence S of each frame; Step 2: Use the spatial feature sequence S of the video as the input of the temporal convolutional network, and output the temporal feature sequence R of the video; Step 3: Using the spatial feature sequence S and temporal feature sequence R of the video as input to the local Transformer module, calculate the local inter-frame correlation and output the temporal feature sequence F incorporating the local inter-frame correlation. (l) ; The local Transformer module only calculates the attention weights between the current frame and its local neighboring frames: MH (l) =Concat(H1 (l) ,H2 (l) ,…,H M (l) ) F (l) =MH (l) +S in This represents the vector corresponding to the i-th node in the value sequence of the m-th head. This represents the vector corresponding to the t-th node in the query sequence of the m-th head. The vector representing the i-th node in the key sequence of the m-th head, nerb(t) represents the neighboring frame; H M (l) h represents the output of the m-th head in the local Transformer. m,t (l) Represents H M (l) The vector corresponding to the t-th node; MH (l) F represents the result of the multi-head attention mechanism. (l) This represents the output of the local Transformer module, where S represents the spatial feature sequence. Step 4: Using the spatial feature sequence S and temporal feature sequence R of the video as input to the global Transformer module, calculate the long-term dependencies between nodes in the global video sequence and output the temporal feature sequence F incorporating the long-term dependencies. (g) ; In steps 3 and 4, the linear transformation of the spatial feature sequence is used as the query value and fed into the local Transformer module and the global Transformer module, respectively. The linear transformation of the temporal feature sequence is used as the key and value and fed into the local Transformer module and the global Transformer module, respectively. Where S represents the spatial feature sequence, R represents the temporal feature sequence, and W... m K W m V W m Q K represents the mapping matrix of the key, value, and query item of the m-th head parameter, respectively. m V m Q m These represent the key, value, and query of the m-th head, respectively. The global Transformer module only calculates the attention weights between the current frame and other frames outside the local neighborhood: MH (g) =Concat(H1 (g) ,H2 (g) ,…,H M (g) ) F (g) =MH (g) +S in This represents the vector corresponding to the i-th node in the value sequence of the m-th head. This represents the vector corresponding to the t-th node in the query sequence of the m-th head. The vector representing the i-th node in the key sequence of the m-th head, nerb(t) represents the neighboring frame, and H M (g) h represents the output of the m-th head in the global Transformer. m,t (g) Represents H M (g) The vector corresponding to the t-th node, MH (g) F represents the result of the multi-head attention mechanism. (g) This represents the output of the global Transformer module, where S represents the spatial feature sequence. Step 5: Incorporate the temporal feature sequence F with local inter-frame correlation. (l) With time-series feature sequences F incorporating long-term dependencies (g) Feature fusion is performed and the data is input into a fully connected network to predict the importance of each frame; Step 6: Train the above network model in a supervised manner using a labeled dataset; divide the original video into segments and input them into the trained model, calculate the importance of each segment, and select the segments with high importance to generate a dynamic summary.

2. The video summarization method based on Transformer and TCN as described in claim 1, characterized in that: In step 1, the pre-trained convolutional network model is the GoogleNet network, and features are extracted through the pre-trained GoogleNet network.

3. The video summarization method based on Transformer and TCN as described in claim 1, characterized in that: In step 3, the method for dividing the local and global regions is a sliding window: nerb(t)={tl,t-l+1,…,t,…,t+l} Where l represents the size of the neighborhood window, and nerb(t) represents the neighborhood of the t-th node.

4. The video summarization method based on Transformer and TCN as described in claim 1, characterized in that: In step 4, the method for dividing the local and global regions is a sliding window: nerb(t)={tl,t-l+1,…,t,…,t+l} Where l represents the size of the neighborhood window, and nerb(t) represents the neighborhood of the t-th node.

5. The video summarization method based on Transformer and TCN as described in claim 1, characterized in that: In step 5, feature fusion is performed by adding features together: F=F (g) +F (l) Where F (g) F represents the output of the global Transformer module. (l) This represents the output of the local Transformer module, and F represents the result of feature fusion.

6. The video summarization method based on Transformer and TCN as described in claim 1, characterized in that: Step 6 involves segmenting the data using the KTS algorithm.

7. The video summarization method based on Transformer and TCN as described in claim 1, characterized in that: In step 6, the 0-1 knapsack algorithm is used to select segments with high importance and generate dynamic summaries.

Images