Automatic video summarization method and system based on self-attention mechanism in cloud environment

Abstract

The application belongs to the field of computer vision, and discloses an automatic video abstracting method and system based on a self-attention mechanism, which relies on original video of a user as input, forms a centralized attention feature matrix through a new frame feature selection method, and obtains video frame scores through a video frame importance scoring model based on a regression network and a self-defined loss function for correction. The method is deployed on a cloud server side, the frame score result is sent to a client after being obtained on the server side, and a video abstract is finally automatically synthesized. The application overcomes the problems of insufficient utilization of attention information, repeated calculation of frame feature information and low timeliness of existing video abstracting methods, and the system is deployed based on a cloud environment, reduces the requirement for hardware performance of the client, realizes end-to-end deployment and training, and finally effectively improves the performance of automatic video abstract generation.

Description

Technical Field

[0001] This invention belongs to the field of video summarization technology in the field of deep learning computer vision, and particularly relates to an automatic summarization method and system based on self-attention mechanism in a cloud environment. Background Technology

[0002] In today's internet age, video has become the most important medium for conveying information in people's work, study, and entertainment. The explosive growth of video data has placed new demands on stable and efficient video storage and retrieval. The main purpose of video summarization is to shorten the length of a video while preserving its core content. Automatic video summarization is a crucial prerequisite for rapid video storage and retrieval, and therefore has broad application prospects. Currently, the main methods for generating video summaries can be divided into two types: supervised and unsupervised techniques.

[0003] Supervised video summarization typically treats video summarization as a structured prediction or sequence labeling problem. It requires a large amount of manually annotated video data, mainly consisting of each video segment and related basic information or importance scores of video frames. Based on the manually annotated information, recurrent neural networks (such as long short-term memory networks) are mainly used to extract the spatiotemporal dependencies between video frames. Then, keyframes are selected according to the video frame scores, and finally, a video summary of the specified length is generated.

[0004] Unsupervised video summarization eliminates the need for manual annotation of video data. Through self-learning, it automatically identifies keyframes after training the original video, thus saving significant labor costs. Unsupervised video summarization uses recurrent networks as the backbone to generate summaries via a summarizer, or generative adversarial networks (GANs), or reinforcement learning methods, utilizing certain characteristics of the summaries to construct a reward function for training the model.

[0005] Based on the above analysis, the problems and shortcomings of existing video summarization technologies are as follows:

[0006] (1) Most solutions use recurrent neural networks to obtain the features and dependencies between video frames. However, recurrent neural networks cannot effectively utilize GPUs for parallel computing, and in practical applications, they cannot effectively extract the dependencies between video frames for long videos.

[0007] (2) The method of generating video summaries based on generative adversarial networks has problems such as unstable training models, high requirements for user terminal performance, and easy mode collapse.

[0008] (3) Existing video summarization methods that use self-attention mechanisms or attention matrices as aids are considered to be the most advanced video summarization methods at present. However, these methods do not fully mine the attention information of video frames and cannot effectively help represent the inter-frame dependencies, such as the uniqueness and diversity of frames, thus affecting the accurate selection of important frames. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention provides an automatic summarization method and system based on a self-attention mechanism in a cloud environment.

[0010] This invention is implemented as follows: an automatic summarization method based on a self-attention mechanism in a cloud environment, specifically including the following steps:

[0011] S1: Obtain the original video information provided by the user; extract the video frame features of the original video into a 1024-dimensional vector, which is used as the input video frame features;

[0012] S2: Calculate the self-attention mechanism matrix of the video frame; use 3 linear layers to calculate the Q, K and V matrices respectively; and multiply the Q matrix and K matrix to obtain the attention mechanism matrix A;

[0013] S3: Calculate the uniqueness of video frames;

[0014] S4: Form the video frame focus attention feature matrix;

[0015] S5: Calculate the importance score of video frames using the frame focus attention feature matrix;

[0016] S6: Calculate two functions, deviation loss and summary length loss, and correct the model performance based on the results.

[0017] Furthermore, in S3, a new method is used to calculate the uniqueness of video frames, namely, to calculate the frame uniqueness entropy based on the principle of masking information from adjacent frames, and then to average the attention uniqueness entropy matrix by row to obtain the video frame uniqueness vector.

[0018] Specifically, for each a in the attention matrix i,j A new calculation formula is proposed to calculate a corresponding e. i,j ,Right now:

[0019]

[0020]

[0021] In formula (1), i and j are the row and column values ​​of the matrix, e i,j Let a be the uniqueness entropy corresponding to the matrix. i,jThe value corresponding to the attention matrix is ​​m, which is the number of adjacent frames to be masked, usually ranging from 0 to 60, where 0 indicates that adjacent frames are not masked, and T is the total number of video frames.

[0022] In formula (2) u i The frame uniqueness at the corresponding position is obtained by averaging the attention entropy.

[0023] Furthermore, the method for forming the frame-focused attention feature matrix in S4 includes:

[0024] The attention matrix A obtained in the second step is transformed again, retaining the information of adjacent frames that were not calculated in the third step and removing the information of other frames calculated in the previous round. The transformed matrix is ​​multiplied by the V matrix to obtain a weighted feature vector. The uniqueness vector of the video frame is then concatenated to the weighted feature vector, making it a 1025-dimensional vector by adding 1 dimension. A linear layer is then used to reduce the dimensionality of the 1025-dimensional feature vector to 1024 dimensions. The 1024-dimensional feature vector is added to the 1024-dimensional feature vector of the pre-trained original video frame to obtain the concentrated attention feature matrix output.

[0025] Furthermore, in S5, the calculation of the video frame importance score specifically involves using a two-layer linear neural network, inserting a drop layer and a layernorm layer, and finally using the sigmoid function to process the linear layer results, mapping the frame concentration attention feature matrix to the (0,1) interval as the basis for importance scoring.

[0026] Furthermore, in step S6, during model calibration, in addition to using the divergence loss function, a new calibration function is also used to assist in the calibration:

[0027] The new correction function utilizes the uniqueness and diversity of frames during model correction, specifically calculating the summary length loss by subtracting the square of the hyperparameters from the average score of all video frames.

[0028]

[0029] Where σ is the summary length regularization coefficient, s n The importance score for the nth frame output by the model;

[0030] Then, the deviation loss is calculated by combining the squared value of the cosine distance between the importance score and the uniqueness of the video frame:

[0031]

[0032] Where u is the uniqueness vector of all video frames in the entire video, and s is the importance score vector of all video frames output by the model.

[0033] Another object of the present invention is to provide a cloud-based automatic summarization system based on a self-attention mechanism that implements the aforementioned automatic summarization method based on a self-attention mechanism in a cloud environment, wherein the cloud-based automatic summarization system based on a self-attention mechanism includes:

[0034] The local client is used to upload raw video data, video segmentation sampling results and video frames, and receive feedback of video frame information processed by the cloud environment server. It automatically splices the video frames with higher scores into a video summary according to their respective sub-shots.

[0035] The cloud environment server is used to deploy an automatic summarization method based on a self-attention mechanism in the cloud environment, and to store the raw video data, video segmentation sampling results and video frames uploaded by users in the cloud, while also processing the video frames.

[0036] Furthermore, the local client specifically includes:

[0037] The video preprocessing module receives user video input, pre-edits and segments the video by shot, and uploads the processed video data and sampled frames to the cloud environment server. The user transmits the packaged video to the video preprocessing module through the module interface. The video preprocessing module uses a motion detection algorithm based on inter-frame difference to segment the video by shot and obtain sampled frames. The segmented video and sampled frames are then transmitted to the cloud environment server through the cloud server.

[0038] The summary generation module is used to receive keyframe information returned by the cloud environment server, including keyframe number and keyframe score information. According to the given length, the user uses the sub-shot with the highest average score to perform video stitching algorithm to stitch the video into the final video summary in chronological order.

[0039] The cloud environment server specifically includes:

[0040] The self-attention mechanism module is used to select feature frames from video frame information using the self-attention mechanism; and to form the frame-focused attention feature matrix using the frame-focused attention feature matrix formation method in the automatic summarization method based on the self-attention mechanism in the cloud environment.

[0041] The regression network model module is used to score the importance of video frames to obtain keyframe score information, and to calibrate the model through unsupervised learning and a calibration function. The keyframe score information is obtained and the model is calibrated using the automatic summarization method based on self-attention mechanism in the cloud environment, which calculates the importance score of video frames and the model calibration method.

[0042] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the automatic summarization method based on a self-attention mechanism in a cloud environment.

[0043] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of automatic summarization based on a self-attention mechanism in the cloud environment.

[0044] Another objective of this invention is to provide an information data processing terminal for implementing an automatic summarization system based on a self-attention mechanism in the cloud environment.

[0045] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0046] First, this invention proposes an automatic video summarization method and system based on a self-attention mechanism in a cloud environment. This method utilizes an advanced self-attention mechanism, focusing on the internal correlation of video frames. It balances the uniqueness and diversity of video frame features in feature selection, while avoiding redundant calculations of frame features. In the scoring model, the loss function is adjusted to automatically correct the model, ensuring full utilization of frame feature information, resulting in more accurate and efficient video summaries. Furthermore, the system is deployed in the cloud, placing core functions on a cloud server, significantly reducing the performance requirements of the client and making it more universal and practical.

[0047] Secondly, the Media Convergence Security and Cognitive Domain Security Team of the School of Cyberspace Security, where this invention is based, has long been engaged in research on video compression and image security, possessing rich theoretical and technical expertise. Based on this experience, an automatic video summarization method and system based on a self-attention mechanism in a cloud environment was designed.

[0048] The research results of this invention can be used to view and search massive video and image resources. Automatic video summarization processing of massive video recordings or real-time images can improve the efficiency of video query, reduce the investment of manpower and material resources, and reduce the performance requirements of personal terminals. At the same time, the research results can provide very beneficial theoretical guidance and practical significance for using video summarization to quickly detect the authenticity of videos and prevent video tampering.

[0049] Third, does the technical solution of this invention overcome technical bias? In this invention, an advanced self-attention mechanism is used to obtain the dependencies between video frames. Compared with previous algorithms, this patent makes full use of the uniqueness and diversity information of video frames to help generate video summaries. It proposes a method to calculate uniqueness by masking the entropy of adjacent frame information in the calculation of frame diversity and uniqueness, which reduces unnecessary repeated calculations, simplifies the calculation process, and greatly improves the calculation performance. It is a more lightweight video frame selection algorithm. At the same time, a new loss function that utilizes frame uniqueness is proposed to help obtain information on frame diversity. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the steps of the automatic video summarization method based on the self-attention mechanism provided in the embodiment of the present invention;

[0051] Figure 2 This is a diagram of an automatic video summarization system based on a cloud environment using a self-attention mechanism, provided in an embodiment of the present invention.

[0052] In the diagram: 1. Local client; 2. Cloud environment server; 3. Video preprocessing module; 4. Summarization module; 5. Attention mechanism module; 6. Regression model module. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] Given the uniqueness and diversity of video frame information, the key to improving the accuracy and efficiency of video summarization is to fully explore the dependencies between frames, select important frames, avoid the inefficiency caused by repeatedly calculating frame features, reduce the performance requirements of the user end, and make full use of the self-attention mechanism for video summarization.

[0055] To address the above problems, this invention provides a novel video automatic summarization method based on a self-attention mechanism. This method fully considers the characteristics of video frame information, accurately and efficiently generating video keyframes, thereby ultimately achieving automatic video summarization. Furthermore, this method leverages a cloud environment to form a complete solution. By utilizing the cloud to complete the keyframe selection for video summarization, the performance requirements of user-end devices are reduced, improving the efficiency and stability of video summarization generation.

[0056] The automatic summarization method based on self-attention mechanism in a cloud environment provided by the embodiments of the present invention mainly includes the following aspects:

[0057] 1) Calculation of the self-attention mechanism matrix: This method uses three linear layers to calculate the Q, K, and V matrices, and then multiplies the Q and K matrices to obtain the attention mechanism matrix A. This self-attention mechanism can capture the interrelationships between video frames, thereby improving the quality and accuracy of the video frame focus attention feature matrix.

[0058] 2) Calculation of video frame uniqueness: This method determines the importance of a video frame in the video summary by calculating its uniqueness. This method can avoid the repetition of similar video frames in the video summary, thereby improving the diversity and readability of the video summary.

[0059] 3) Method for calculating video frame importance scores: This method calculates the importance score of a video frame using the frame focus attention feature matrix. This method can calculate the importance score of a video frame more accurately while ensuring the quality and accuracy of the video frame focus attention feature matrix.

[0060] 4) Calculation of Deviation Loss and Summary Length Loss: This method corrects the model's performance by calculating two functions: deviation loss and summary length loss. This method can effectively avoid video summarization being too simple or too complex, thereby improving the readability and usability of video summaries.

[0061] The automatic video summarization method based on self-attention mechanism in the cloud environment described in this application consists of two parts: a video frame feature selection method and a video frame importance scoring model based on a regression network. First, the video frame feature vectors are filtered, and then the video frames are scored for importance.

[0062] The following is a further explanation in conjunction with the accompanying drawings:

[0063] like Figure 1 As shown, the automatic summarization method based on self-attention mechanism in the cloud environment specifically includes the following steps:

[0064] This invention provides an automatic video summarization method based on a self-attention mechanism, comprising the following steps:

[0065] S101, Obtain the original video information provided by the user;

[0066] The self-attention module receives a 1024-dimensional video frame feature vector output by the feature extraction module. Specifically, the number of video frames is n, and an n×1024 matrix is ​​received.

[0067] S102, Calculate the self-attention mechanism matrix of the video frame;

[0068] This feature matrix is ​​passed through three linear layers with different parameters to obtain the Q, K, and V matrices. The attention matrix A is obtained by multiplying the Q and K matrices.

[0069] S103, Calculate the uniqueness of video frames;

[0070] An attention uniqueness entropy matrix is ​​calculated using attention matrix A. The video frame uniqueness vector is obtained by averaging the row-wise values ​​of the attention uniqueness entropy matrix. Specifically, each a in attention matrix A... i,j Calculate a corresponding e i,j Use formula

[0071]

[0072]

[0073] Where m is the number of adjacent frames to be blocked (the value can be between 0 and 60), here we can choose m = 30, u i For the frame uniqueness of the corresponding position;

[0074] S104, forming the video frame focus attention feature matrix;

[0075] Transform the attention matrix A to remove frames calculated in the previous round, while retaining the information of frames masked in the previous round. Specifically, as shown in the following pseudocode:

[0076] for(i=1,i<=n,i++)

[0077] for(j=1,j<=n,j++)

[0078] if (j>i+m&&j) <i-m)

[0079] a i,j =0

[0080] Multiply the transformed matrix by the V matrix to obtain a weighted eigenvector. Then, concatenate the unique eigenvectors of the video frames to the weighted eigenvector, making it a 1025-dimensional vector by adding one dimension from a 1024-dimensional vector. The matrix of all video frame features is n×1025.

[0081] By passing an n×1025 matrix through a linear layer, the 1025-dimensional eigenvectors are reduced to 1024 dimensions, becoming an n×1024 matrix.

[0082] The n×1024 matrix is ​​added to the n×1024 matrix of the original video frame features given by the feature extraction module to obtain the output of the self-attention module, which is an n×1024 matrix.

[0083] S105, calculate the importance score of the video frame using the attention feature matrix of the feature frame information;

[0084] The obtained self-attention matrix is ​​subjected to dropout and layernorm operations, then fed into a linear layer, and non-linear mapping is performed using the ReLU function. The features are then subjected to dropout and layernorm operations again, then fed into a linear layer, and non-linear mapping is performed using the Sigmoid function. Finally, the importance score of each video frame is obtained, with the importance score value in the interval (0,1).

[0085] S106. After calculating the importance scores, two loss functions need to be calculated.

[0086] Calculate the values ​​of two loss functions, namely, the summary length loss. and deviation loss Where u is the uniqueness vector of all video frames in the entire video, and s is the importance score vector of all video frames output by the model.

[0087] At the same time Backpropagation is used to compute gradients and update the corresponding parts of the architecture.

[0088] like Figure 2 As shown in the figure, the automatic video summarization system structure with self-attention mechanism in a cloud environment provided by this embodiment of the invention consists of two parts: a local client 1 and a cloud environment server 2.

[0089] Local client 1 includes a video preprocessing module and a summary generation module.

[0090] The video processing module of local client 1 receives the user's settings for video summarization and preprocessing the segmented video. The summarization module receives information such as keyframe scores returned from the cloud. The user performs preliminary editing (e.g., removing the intro) and cutting of the video to be summarized according to their own needs, and inputs the length of the video summary. The local client uses a video sampling algorithm to sample the video determined by the user at a rate of 2fps, changes the resolution to 300x300, and packages the compressed video data and sends it to the server.

[0091] On the cloud server side, the main tasks are video frame feature selection and frame importance scoring. Initially, the model is started on the cloud server to obtain the trained model architecture and parameters.

[0092] After deployment, the server first uses a public dataset and a large number of other collected video samples to extract video frame features using a pre-trained network, or some pre-extracted video frame features, and then randomly groups them.

[0093] Randomly initialize model parameters, set model training stopping conditions, and stop training the empirical model after 200 rounds.

[0094] Set the hyperparameters for model training, typically with a batch size of 16 (features from all frames of 16 videos), a masking number of adjacent frames of 30, and a summary length regularization factor of 0.7.

[0095] The features of a batch of video frames in the training set are randomly selected, and the total loss is calculated by applying this feature to the loss function of the model during training.

[0096] The loss calculated using the trained model is used to update the entire model parameters using the backpropagation algorithm. Specifically, the loss is used to calculate the gradient of the parameters for each neural network layer using chain derivatives, and the gradient descent algorithm is used to update the model parameters.

[0097] This process is repeated continuously until 200 rounds of training are completed, at which point the trained model parameters are uploaded to the model inference module, which then updates the model parameters.

[0098] After obtaining the importance score of each video frame, local client 1 calculates the importance of each video segment using the KTS algorithm. Specifically, the importance of a video segment is the average of the importance scores of all the video frames it contains. The video segments are then concatenated from highest to lowest importance until the preset video summary length is met.

[0099] The automatic video summarization method based on self-attention mechanism in a cloud environment provided in the application embodiment of the present invention is applied to a computer device, the computer device including a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor performs the steps of the automatic video summarization method based on self-attention mechanism in a cloud environment.

[0100] The automatic video summarization method based on self-attention mechanism in a cloud environment provided by the application embodiment of the present invention is applied to an information data processing terminal, which is used to implement the automatic video summarization system based on self-attention mechanism in the cloud environment.

[0101] The embodiments of the present invention have achieved some positive results during the research and development or use process, and have indeed great advantages compared with the prior art. The following content describes them in conjunction with the data, charts and other information of the experimental process.

[0102] The method proposed in this patent was applied to the SumMe and TVSum datasets. For the SumMe dataset, a common method was used to calculate the f-score metric by considering the similarity between machine-generated and user-defined summaries for a given video. For the TVSum dataset, a common method was used to convert frame-level annotations into key-fragment annotations, and then the generated summaries were compared with available user summaries for the video, calculating the F-score for each pair of compared summaries. The results are as follows:

[0103] TABLE I: Performanc comparisons

[0104]

[0105] Compared to stochastic parameter models, the method proposed in this invention (last line) is better able to compute the uniqueness of video frames on the SumMe dataset, and can be used to improve the performance of video summarization models. It also shows substantial improvements on the TVSum dataset, while remaining competitive.

[0106] The automatic summarization system based on self-attention mechanism in a cloud environment provided in this embodiment of the invention includes two parts: a local client and a cloud environment server.

[0107] In the local client, the video preprocessing module receives user video input, pre-edits and segments the video by shot, and uploads the processed video data and sampled frames to the cloud environment server. Specifically, the user transmits the pre-processed video, packaged and sent to the video preprocessing module through the module interface. The video preprocessing module uses a motion detection algorithm based on inter-frame difference to segment the video by shot and obtains sampled frames. The segmented video and sampled frames are then transmitted to the cloud environment server via the cloud server.

[0108] In the cloud environment server, the self-attention mechanism module utilizes the self-attention mechanism to select feature frames from video frame information. It then uses the frame-focused attention feature matrix formation method from the cloud-based self-attention mechanism-based automatic summarization method to form the frame-focused attention feature matrix. The regression network model module scores the importance of video frames to obtain keyframe score information. It then uses unsupervised learning and a correction function to calibrate the model, employing the video frame importance score calculation method and model calibration method from the cloud-based self-attention mechanism-based automatic summarization method to obtain keyframe score information and calibrate the model.

[0109] Finally, the summary generation module receives keyframe information returned by the cloud environment server, including keyframe number and keyframe score information. Based on the given length, the user uses the sub-shot with the highest average score to perform a video stitching algorithm to stitch the video into the final video summary in chronological order.

[0110] The specific working principle is as follows: First, on the local client, the video preprocessing module preprocesses and segments the user-input video by shot, and uploads the processed video data and sampled frames to the cloud environment server. Then, on the cloud environment server, the self-attention mechanism module uses the self-attention mechanism to select feature frames from the video frame information, forming a frame-focused attention feature matrix. The regression network model module scores the importance of video frames to obtain keyframe score information, and corrects the model through unsupervised learning and a correction function. Finally, the summarization module receives the keyframe information returned from the cloud environment server, and according to the given length, uses the sub-shot with the highest average score to execute a video stitching algorithm to stitch together the final video summary. Through the coordinated work of the above steps, the system can achieve automatic summarization based on the self-attention mechanism.

[0111] The following are six embodiments provided by this invention, demonstrating the application of the automatic summarization method based on the self-attention mechanism in different fields:

[0112] 1) Video surveillance field: Applying this method to the video surveillance field can automatically identify and summarize important events in the video, making it convenient for security personnel to quickly browse and analyze.

[0113] 2) News summarization: Applying this method to the news summarization field can automatically extract key information from a large number of news reports and generate concise and accurate news summaries, making it convenient for users to quickly understand the key points of the news.

[0114] 3) Film editing: Applying this method to film editing can automatically extract important plot points and shots from a film and perform intelligent editing, thereby improving film production efficiency.

[0115] 4) Meeting minutes: Applying this method to meeting minutes can automatically extract important information from meeting recordings and notes, and generate concise and accurate meeting summaries, making it easy for participants to quickly review the key points of the meeting.

[0116] 5) Academic Papers: Applying this method to academic papers can automatically extract key information from a large number of academic papers and generate concise and accurate paper abstracts, making it easy for readers to quickly understand the paper's topic and research results.

[0117] 6) Social Media Domain: Applying this method to the social media domain can automatically extract key information from a large number of social media posts and generate concise and accurate summaries, making it easy for users to quickly understand the hot topics and discussion trends on social media.

[0118] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An automatic summarization method based on self-attention mechanism in a cloud environment, characterized in that, A self-attention mechanism is used to calculate the attention feature matrix and importance score of video frames. At the same time, the model performance is corrected by calculating two functions: deviation loss and summary length loss, thereby improving the quality and accuracy of video summarization. Includes the following steps: S1: Obtain the original video information provided by the user; extract the video frame features of the original video into a 1024-dimensional vector, which is used as the input video frame features; S2: Calculate the self-attention mechanism matrix of the video frame; use 3 linear layers to calculate the Q, K and V matrices respectively; and multiply the Q matrix and K matrix to obtain the attention mechanism matrix A; S3: Calculate the uniqueness of video frames; S4: Form the video frame focus attention feature matrix; S5: Calculate the importance score of video frames using the frame focus attention feature matrix; S6: Calculate the two functions, deviation loss and summary length loss, and correct the model performance based on the results; In S3, when calculating the uniqueness of a video frame, the frame uniqueness entropy is calculated based on the principle of masking information from adjacent frames. Then, the attention uniqueness entropy matrix is ​​averaged row by row to obtain the video frame uniqueness vector. Specifically, for each a in the attention mechanism matrix A i,j A new calculation formula is proposed to calculate a corresponding e. i,j ,Right now: In formula (1), i and j are the row and column values ​​of the matrix, e i,j Let a be the uniqueness entropy corresponding to the matrix. i,j is the value corresponding to the attention matrix, m is the number of adjacent frames to be masked, ranging from 0 to 60, where 0 indicates that adjacent frames are not masked, and T is the total number of video frames; In formula (2) u i The frame uniqueness at the corresponding position is obtained by averaging the attention entropy; The method for forming the frame-focused attention feature matrix in S4 includes: The attention mechanism matrix A obtained in the second step is transformed again, retaining the adjacent frame information that was not calculated in the third step and removing the other frame information calculated in the previous round. The transformed matrix is ​​multiplied by the V matrix to obtain a weighted feature vector. The uniqueness vector of the video frame is then concatenated to the weighted feature vector, making it a 1025-dimensional vector by adding 1 dimension. A linear layer is then used to reduce the dimensionality of the 1025-dimensional feature vector to 1024 dimensions. The 1024-dimensional feature vector is added to the 1024-dimensional feature vector of the pre-trained original video frame to obtain the concentrated attention feature matrix output. The video frame importance score is calculated in S5 by using a two-layer linear neural network with a drop layer and a layernorm layer inserted in between. Finally, the sigmoid function is used to process the linear layer results, and the frame concentration attention feature matrix is ​​mapped to the (0,1) interval as the basis for importance scoring. In step S6, during model calibration, in addition to using the divergence loss function, a new calibration function is also used to assist in the calibration: During model calibration, the uniqueness and diversity of frames are utilized, specifically by subtracting the square of the hyperparameters from the average score of all video frames to calculate the summary length loss. Where σ is the summary length regularization coefficient, s n The importance score for the nth frame output by the model; Then, the deviation loss is calculated by combining the squared value of the cosine distance between the importance score and the uniqueness of the video frame: Where y is the uniqueness vector of all video frames in the entire video, and s is the importance score vector of all video frames output by the model.

2. A cloud-based automatic summarization system based on a self-attention mechanism, implementing the automatic summarization method based on a self-attention mechanism in a cloud environment as described in claim 1, characterized in that, The automatic summarization system based on self-attention mechanism in the cloud environment includes: The local client is used to upload raw video data, video segmentation sampling results and video frames, and receive feedback of video frame information processed by the cloud environment server. It automatically splices the video frames with higher scores into a video summary according to their respective sub-shots. The cloud environment server is used to deploy an automatic summarization method based on a self-attention mechanism in the cloud environment, and to store the raw video data, video segmentation sampling results and video frames uploaded by users in the cloud, while also processing the video frames.

3. The automatic summarization system based on self-attention mechanism in a cloud environment as described in claim 2, characterized in that, The local client specifically includes: The video preprocessing module receives user video input, pre-edits and segments the video by shot, and uploads the processed video data and sampled frames to the cloud environment server. The user submits the pre-processed video, segments the video by shot, packages and labels it, and transmits the packaged video to the video preprocessing module through the module interface. The video preprocessing module uses a motion detection algorithm based on inter-frame difference to segment the video by shot and obtains sampled frames. The segmented video and sampled frames are then transmitted to the cloud environment server through the cloud server. The summary generation module is used to receive keyframe information returned by the cloud environment server, including keyframe number and keyframe score information. According to the given length, the user uses the sub-shot with the highest average score to perform video stitching algorithm to stitch the video into the final video summary in chronological order. The cloud environment server specifically includes: The self-attention mechanism module is used to select feature frames from video frame information using the self-attention mechanism; and to form the frame-focused attention feature matrix using the frame-focused attention feature matrix formation method in the automatic summarization method based on the self-attention mechanism in the cloud environment. The regression network model module is used to score the importance of video frames to obtain keyframe score information, and to calibrate the model through unsupervised learning and a calibration function. The keyframe score information is obtained and the model is calibrated using the automatic summarization method based on self-attention mechanism in the cloud environment, which calculates the importance score of video frames and the model calibration method.

4. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the automatic summarization method based on a self-attention mechanism in a cloud environment as described in claim 1.

5. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the automatic summarization system based on self-attention mechanism in the cloud environment as described in claims 2-3.

Images