Unsupervised video summarization method and system based on hybrid twin mask autoencoder

Abstract

The application discloses an unsupervised video abstracting method and system based on a hybrid twin mask automatic encoder, the method directly quantifies the importance of each shot in the video through the mode of restoring the shot after the mask, improves the accuracy of shot importance evaluation, is superior to other advanced algorithms in the fitting result of artificial abstracting, effectively solves the unstable training problem of the previous unsupervised video abstracting method based on the generative adversarial model, makes the generated abstracting result more stable, and the model training of the method is not dependent on complex artificial annotation, and compared with the supervised method, has stronger feasibility, and can be effectively used in the fields of video classification, retrieval and the like.

Description

Technical Field

[0001] This invention belongs to the field of video summarization technology, and relates to an unsupervised video summarization method and system based on a hybrid twin mask autoencoder. Background Technology

[0002] With the development of video sharing platforms and the widespread use of mobile devices, the number of videos circulating online has exploded. The need for rapid retrieval, understanding, and management of this massive amount of video content has become urgent. Video summarization technology is a key technology to meet this need. It utilizes video analysis techniques to select representative portions from raw video data to generate concise summaries, allowing users to quickly understand the main content of the original video. Video summarization methods mainly include two forms: static video summarization based on keyframes and dynamic video summarization based on key shots.

[0003] In recent years, with the rise of deep learning technology, research methods using deep learning have been widely applied in video summarization research. Most video summarization methods first use the KTS algorithm to segment the video into shots, then predict the frame importance score in each shot, and combine these scores to determine the shot importance score. Finally, the 0-1 knapsack algorithm is used to extract key shots as the video summary. However, treating shot importance scores as a set of corresponding frame importance scores may be inaccurate without fully utilizing the temporal and spatial contextual semantics of the video.

[0004] Furthermore, supervised methods train models to identify important frames or shots by using raw videos and manually created summary annotations as training data. However, labeling a large number of manually generated annotations and importance scores is a time-consuming and laborious process to build a video summarizing model with high generalization ability, which limits the practical application of supervised methods. Weakly supervised methods often promote model training by introducing more readily available prior information, but the use of prior information may lead to model bias problems. In addition, different users have different preferences for summaries, and annotators may have different perspectives. Unsupervised and self-supervised methods have shown greater practical application value because they are free from the dependence on large-scale manual annotations. However, self-supervised methods usually require careful design of appropriate agent tasks. Existing unsupervised methods rely heavily on generative adversarial networks, which suffer from unstable model training, resulting in low stability of the generated video summarizing results and low accuracy of shot importance assessment. Summary of the Invention

[0005] The purpose of this invention is to solve the problems in the prior art and provide an unsupervised video summarization method and system based on a hybrid twin mask autoencoder.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] An unsupervised video summarization method based on a hybrid twin mask autoencoder includes the following steps:

[0008] The original video is framed to obtain a video frame sequence. Features are extracted from the video frame sequence to obtain the original frame feature sequence. The shot boundaries are delineated based on the original frame feature sequence.

[0009] Mask the lenses according to the defined lens boundaries to obtain the remaining unmasked lens subset. Recover the masked lenses in the unmasked lens subset based on the original frame feature sequence, obtain the recovery features of the masked lenses, and calculate the mean square error loss between the recovery features and the original frame feature sequence.

[0010] Obtain the shot-level features of the original frame feature sequence, reconstruct the shot-level features and obtain the diversity coefficient of each shot, obtain the diverse shots based on the diversity coefficient, and obtain the repulsion loss of the diverse shots.

[0011] The importance score of each shot is calculated based on the mean squared error loss and the repulsion loss. Key shots are selected based on the importance scores, and the final video summary is generated from the selected key shots.

[0012] A further improvement of the present invention is that:

[0013] The restoration of masked shots from an unmasked shot subset includes the following steps:

[0014] The unmasked shot subset and the original frame feature sequence are encoded using a twin encoder;

[0015] Using the encoded original frame feature sequence as a priori reference, the decoder recovers the masked shots in the subset of unmasked shots.

[0016] The mean squared error loss of the restored features and the original frame feature sequence is calculated using formula (1):

[0017]

[0018] Where, N f F represents the number of frames contained in the masked shot subset S′. R The masked lens features representing the model recovery; F V This represents the features of the extracted frame.

[0019] Acquiring diverse shots involves the following steps:

[0020] For the encoded original frame feature sequence F PLens-level features are obtained by averaging the features of all frames within the shot;

[0021] Encoding the lens-level features yields weighted lens features.

[0022] For lens features F S Feature reconstruction is performed separately, and the diversity coefficient of each shot is given;

[0023] The lens diversity coefficient is used to select the reconstruction features of several lenses and calculate their rejection loss.

[0024] The repulsion loss of the diverse lenses is calculated using formula (2):

[0025] F′ S =Top-v(φ(F) S ),y(F S ),v=Round(M×∈))

[0026]

[0027] The Top-v() function represents the selection of the diversity coefficient y(F). S The reconstructed shot feature φ(F) corresponding to the largest v shots in ) S ), ∈ are hyperparameters for selecting lenses.

[0028] The process of generating the final video summary includes the following steps:

[0029] The reconstruction score of each shot is multiplied by the corresponding diversity coefficient to obtain the final importance score of the shot. Combined with the 0-1 knapsack algorithm, key shots are selected to generate the final video summary.

[0030] The reconstruction score of the lens is obtained by formula (3):

[0031]

[0032] Where, N l It represents the number of frames contained in each shot, and the shot diversity coefficient is directly generated by the fractional regression layer.

[0033] An unsupervised video summarization system based on a hybrid twin mask autoencoder includes:

[0034] The original video processing module is used to extract frames from the original video to obtain a video frame sequence, extract features from the video frame sequence to obtain an original frame feature sequence, and delineate the shot boundaries based on the original frame feature sequence.

[0035] The mean squared error loss calculation module is used to mask the lens according to the divided lens boundaries, obtain the remaining unmasked lens subset, recover the masked lens in the unmasked lens subset based on the original frame feature sequence, obtain the recovery features of the masked lens, and calculate the mean squared error loss between the recovery features and the original frame feature sequence.

[0036] The repulsion loss calculation module is used to obtain the shot-level features of the original frame feature sequence, reconstruct the shot-level features and obtain the diversity coefficient of each shot, obtain the diverse shots based on the diversity coefficient, and obtain the repulsion loss of the diverse shots.

[0037] The video summarization module is used to calculate the importance score of each shot based on mean squared error loss and repulsion loss, select key shots based on the importance scores, and generate the final video summary using the selected key shots.

[0038] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in this invention.

[0039] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in this invention.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This invention discloses an unsupervised video summarization method based on a hybrid twin mask autoencoder. This method directly quantifies the importance of each shot in the video by restoring the shot after masking, improving the accuracy of shot importance assessment. Compared with the fitting results of human summarization, it outperforms other existing advanced algorithms. It effectively solves the problem of unstable training in previous unsupervised video summarization methods based on generative adversarial models, making the generated summarization results more stable. Moreover, the model training of this method does not rely on complex manual annotations, making it more feasible than supervised methods and effectively applicable to video classification, retrieval and other fields. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a structural diagram of the model of the present invention;

[0044] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0046] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0047] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0048] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0049] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0050] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0051] The present invention will now be described in further detail with reference to the accompanying drawings:

[0052] See Figures 1 to 2 This invention discloses an unsupervised video summarization method and system based on a hybrid twin mask autoencoder. This method enables stable training of the summarization model without manual annotation, and directly quantifies the importance of each shot in the video to produce more reasonable and accurate video summarization results. The invention demonstrates superior performance on the video summarization datasets SumMe and TVSum, validating its superior performance and practical value.

[0053] Specifically, the following steps are included:

[0054] Step 1: Extract frames from the original video to obtain a video frame sequence, and use the neural network model GoogLeNet to extract features from the frame sequence. Then, delineate the shot boundaries and add positional encoding information.

[0055] Specifically:

[0056] The original video was downsampled at a rate of 2 frames per second to obtain a downsampled video. Then, the pre-trained GoogLeNet was used to extract the features of the frames. Where T is the number of frames after video sampling, and C is the dimension of the extracted frame features;

[0057] The transition points of the original video are extracted from the frame features using a kernel temporal segmentation algorithm, which serve as the boundaries for shot division.

[0058] Adding sinusoidal position coding information to the frame features yields a frame feature sequence with added position coding.

[0059] Step 2: The multi-view twin masking autoencoder masks the shots with a fixed step size according to the divided shot boundaries, obtaining the remaining unmasked shot subsets. These subsets are input into the twin encoder for feature encoding. At the same time, the original frame feature sequence is also encoded by the twin encoder with shared weights. The decoder uses the prior reference information of the original frame feature sequence to try to recover the masked shots in the unmasked shot subset. The reconstruction score of each shot is obtained by calculating the mean square error loss between the recovered features and the original frame feature sequence.

[0060] Specifically:

[0061] After masking the frame sequence according to the shot boundaries with a fixed step size, an unmasked subset of shots is obtained;

[0062] The unmasked shot subset and the original frame feature sequence are encoded using a twin encoder;

[0063] Using the encoded original frame feature sequence as a priori reference, the masked shots in the shot subset are recovered by the decoder.

[0064] The method for obtaining the unmasked shot subset: For the frame feature sequence F′ with added position encoding... V Based on the defined shot boundaries, the set of shots can be represented as S = [S1, S2, ..., S...]. i S M ], where M is the number of shots divided. By masking S N times with a fixed step size N (N>1), N unmasked shot subsets can be obtained. The remaining unmasked shot subset S after the k-th (k∈{1,2,…,N}) masking is... k for:

[0065]

[0066] S k =SS′

[0067] In this embodiment, the twin encoder is constructed using a GL-Transformer module that employs global and local attention, and it processes the input N unmasked lens subsets S. k and the complete frame feature sequence F′ V The encoding results are obtained by modeling global and local dependencies respectively. and Where T k S represents the subset of unmasked lenses. k The number of frames included.

[0068] The decoder is built upon GL-Transformer blocks and cross-attention, encoding the complete frame feature sequence F using a Siamese encoder. PFor the keys and values ​​of cross-attention, the unmasked subset of shots encoded by a twin encoder Cross-attention is used as the query, and then the lens of the mask is restored through the GL-Transformer module.

[0069] The GL-Transformer block combines Global Multi-Head Self-Attention (GMSA) and Local Multi-Head Self-Attention (LMSA) to perform multi-scale dependency modeling on the global and local aspects of the input sequence, for an input G (including N unmasked shot subsets S). k Given a complete set of lenses (S), the computation process of the GL-Transformer module is as follows:

[0070]

[0071]

[0072] H i =Att(Q i K i V i (7)

[0073] LMSA = Concat(H1, H2, ..., H h W O (8)

[0074] In the formula, It is a learnable parameter matrix, Q i K i V i These are the calculated query matrix, key matrix, and value matrix, respectively. Att() is the attention function, h is the number of heads for local attention, and A is the mask matrix. When frames i and j are in the same segment, A... ij =1, otherwise A ij =0, ⊙ denotes the Hadamard product of the matrix. H i These are the intermediate features extracted by each attention head, and Concat(·) is the connection operation.

[0075] Step 3: The lens diversity enhancer obtains lens-level features by aggregating the original frame feature sequences encoded by the twin encoder; then the lens diversity enhancer reconstructs the lens features using a decoder, fractional regression layer and linear embedding layer and gives the diversity coefficient of each lens; based on the diversity coefficient, the lenses with diversity are selected and their repulsion loss is calculated.

[0076] Specifically:

[0077] The original frame feature sequence F encoded by the encoder P Lens-level features are obtained by averaging the features of all frames within the shot;

[0078] The lens diversity enhancer uses a decoder built on a Transformer module to encode lens-level features to obtain weighted lens features.

[0079] The fractional regression layer and the linear embedding layer are fully connected layers, which are used to analyze the lens features F. S Feature reconstruction is performed separately, and the diversity coefficient for each shot is given. Based on the shot diversity coefficient, the reconstructed features of several shots are selected, and their repulsion loss is calculated to enhance the diversity of shots included in the abstract.

[0080] Step 4: The model is trained unsupervised based on mean squared error loss and repulsion loss. The reconstruction score of each shot is multiplied by the corresponding diversity coefficient to obtain the final importance score of the shot. Combined with the 0-1 knapsack algorithm, key shots are selected to generate the final video summary.

[0081] Using the mean square error loss of the reconstructed lens obtained in step 2) and the repulsion loss of the lens obtained in step 3), a loss hyperparameter is set to balance the influence of the two losses. The network parameters are continuously updated through backpropagation.

[0082] The mean square error of the reconstructed lens is expressed as:

[0083]

[0084] Where, N f F represents the number of frames contained in the masked shot subset S′. R The mask lens features representing the model's reconstruction;

[0085] The calculation process for the repulsion loss is as follows:

[0086] F′ S =Top-v(φ(F) S ),y(F S ),v=Round(M×∈))

[0087]

[0088] The Top-v() function represents the selection of the diversity coefficient y(F). S The reconstructed shot feature φ(F) corresponding to the largest v shots in ) S ), ∈ are hyperparameters for selecting lenses.

[0089] The total loss function is expressed as:

[0090]

[0091] Among them, α is a hyperparameter that balances the values ​​of the two loss functions.

[0092] The final score of each shot is obtained by multiplying the reconstruction score of the shot obtained in step 2) and the diversity coefficient of the shot obtained in step 3). The key shots are selected as the final summary by combining the 0-1 knapsack algorithm.

[0093] The reconstruction score of the lens is:

[0094]

[0095] Where, N l It is the number of frames contained in each shot, and the diversity coefficient of the shot is directly generated by the fractional regression layer.

[0096] To verify the accuracy and reliability of this invention, it was tested on the video summarization datasets TVSum and SumMe, the F-score was calculated, and the results were compared with existing algorithms. The results are shown in Table 1.

[0097] Under standard settings, the method of this invention outperforms state-of-the-art methods by 8.4% and 1.9% on the SumMe dataset and the TvSum dataset, respectively.

[0098] Table 1. F-score test results on the SumMe and TvSum datasets.

[0099]

[0100] This embodiment discloses an unsupervised video summarization system based on a hybrid twin mask autoencoder, including:

[0101] The original video processing module is used to extract frames from the original video to obtain a video frame sequence, extract features from the video frame sequence to obtain an original frame feature sequence, and delineate the shot boundaries based on the original frame feature sequence.

[0102] The mean squared error loss calculation module is used to mask the lens according to the divided lens boundaries, obtain the remaining unmasked lens subset, recover the masked lens in the unmasked lens subset based on the original frame feature sequence, obtain the recovery features of the masked lens, and calculate the mean squared error loss between the recovery features and the original frame feature sequence.

[0103] The repulsion loss calculation module is used to obtain the shot-level features of the original frame feature sequence, reconstruct the shot-level features and obtain the diversity coefficient of each shot, obtain the diverse shots based on the diversity coefficient, and obtain the repulsion loss of the diverse shots.

[0104] The video summarization module is used to calculate the importance score of each shot based on mean squared error loss and repulsion loss, select key shots based on the importance scores, and generate the final video summary using the selected key shots.

[0105] A schematic diagram of a terminal device according to an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0106] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0107] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0108] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0109] The memory can be used to store the computer program and / or module. The processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0110] If the modules / units integrated into the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0111] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An unsupervised video summarization method based on a hybrid twin mask autoencoder, characterized in that, Includes the following steps: The original video is framed to obtain a video frame sequence. Features are extracted from the video frame sequence to obtain the original frame feature sequence. The shot boundaries are delineated based on the original frame feature sequence. Mask the lenses according to the defined lens boundaries to obtain the remaining unmasked lens subset. Recover the masked lenses in the unmasked lens subset based on the original frame feature sequence, obtain the recovery features of the masked lenses, and calculate the mean square error loss between the recovery features and the original frame feature sequence. Obtain the shot-level features of the original frame feature sequence, reconstruct the shot-level features and obtain the diversity coefficient of each shot, obtain the diverse shots based on the diversity coefficient, and obtain the repulsion loss of the diverse shots. The importance score of each shot is calculated based on the mean squared error loss and the repulsion loss. Key shots are selected based on the importance scores of the shots, and the final video summary is generated from the selected key shots. The restoration of masked shots from an unmasked shot subset includes the following steps: The unmasked shot subset and the original frame feature sequence are encoded using a twin encoder; Using the encoded original frame feature sequence as a priori reference, the decoder recovers the masked shots in the subset of unmasked shots.

2. The unsupervised video summarization method based on a hybrid twin mask autoencoder according to claim 1, characterized in that, The mean squared error loss of the restored features and the original frame feature sequence is calculated using formula (1): in, For a subset of masked shots Number of frames included The mask lens features representing the model's reconstruction; This represents the features of the extracted frame.

3. The unsupervised video summarization method based on a hybrid twin mask autoencoder according to claim 1, characterized in that, Acquiring diverse shots involves the following steps: The original frame feature sequence for encoding Lens-level features are obtained by averaging the features of all frames within the shot; Encoding the lens-level features yields weighted lens features. ; Lens features Feature reconstruction is performed separately, and the diversity coefficient of each shot is given; The lens diversity coefficient is used to select the reconstruction features of several lenses and calculate their rejection loss.

4. The unsupervised video summarization method based on a hybrid twin mask autoencoder according to claim 1, characterized in that, The repulsion loss of the diverse lenses is calculated using formula (2): in, The function represents the choice diversity coefficient. The largest Reconstructed lens features corresponding to each shot , It is a hyperparameter for selecting lenses.

5. The unsupervised video summarization method based on a hybrid twin mask autoencoder according to claim 1, characterized in that, The process of generating the final video summary includes the following steps: The reconstruction score of each shot is multiplied by the corresponding diversity coefficient to obtain the final importance score of the shot. Combined with the 0-1 knapsack algorithm, key shots are selected to generate the final video summary.

6. The unsupervised video summarization method based on a hybrid twin mask autoencoder according to claim 5, characterized in that, The reconstruction score of the lens is obtained by formula (3): in, It represents the number of frames contained in each shot, and the shot diversity coefficient is directly generated by the fractional regression layer.

7. An unsupervised video summarization system based on a hybrid twin mask autoencoder that implements the method of claim 1, characterized in that, include: The original video processing module is used to extract frames from the original video to obtain a video frame sequence, extract features from the video frame sequence to obtain an original frame feature sequence, and delineate the shot boundaries based on the original frame feature sequence. The mean squared error loss calculation module is used to mask the lens according to the divided lens boundaries, obtain the remaining unmasked lens subset, recover the masked lens in the unmasked lens subset based on the original frame feature sequence, obtain the recovery features of the masked lens, and calculate the mean squared error loss between the recovery features and the original frame feature sequence. The repulsion loss calculation module is used to obtain the shot-level features of the original frame feature sequence, reconstruct the shot-level features and obtain the diversity coefficient of each shot, obtain the diverse shots based on the diversity coefficient, and obtain the repulsion loss of the diverse shots. The video summarization module is used to calculate the importance score of each shot based on mean squared error loss and repulsion loss, select key shots based on the importance scores, and generate the final video summary using the selected key shots.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images