A video summarization method based on regret minimization

Abstract

The present application relates to a video summary method, in particular, a method for obtaining a summary by using regret minimization and cutting part of video pictures is designed.The present application is committed to solving the cost problem required when generating a video summary, such as the learning and training cost in the deep learning method or the inquiry cost of the audience, etc.The present application uses a series of image processing algorithms and frameworks to realize the quantification of the information of each picture in the video, so that the picture is converted into multidimensional data, and for a large amount of multidimensional data, the regret minimization algorithm is combined with the video summary task to screen out the summary result, and the result is supplemented in the plot coverage.The present application is widely applicable to various video browsing, video retrieval, video promotion, monitoring query and other application scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for generating a corresponding summary from a video. Specifically, a video summary generation method with multi-attribute constraints based on regret minimization is designed. BACKGROUND

[0002] With the continuous emergence of video information sources and the rapid development of short video platforms, people are increasingly inclined to quickly obtain information through shorter videos. At the same time, with the development of video recording devices such as smartphones, the threshold for video production is constantly lowered, and everyone can share their own recorded videos on the Internet, which makes people often face a large number of videos of varying quality. When the video is very long, it becomes extremely difficult to quickly understand the content of each video and select the content they want. In order to reduce the cost of browsing videos and quickly display the content of long videos, video summary technology has become a hot research topic.

[0003] The goal of video summary is to remove the redundant parts of the video and extract the most important, most relevant, and most interesting parts to form a short summary that reflects the main content of the video. According to the form of the summary, it can be divided into static summary and dynamic summary. Static summary is presented in the form of a picture collection, while dynamic summary is presented in the form of a video clip collection. Static summary can reduce the browsing cost, while dynamic summary can retain the sound and dynamic information of the video. The present application presents in the form of static summary [1, 2, 3].

[0004] Due to the complexity of the video, this technology still has many problems. First, the extraction of video content information is a difficult task. Second, the standard for evaluating a video summary is highly subjective and varies from person to person, making it difficult to objectively evaluate.

[0005] In terms of picture information quantization, some research methods directly use pixel data of video pictures as the basis for calculation, such as clustering, dictionary learning methods, and some deep learning methods. However, the actual information contained in the picture is diverse and can be further separated and extracted. Researchers usually extract low-order features of video pictures such as color histogram features, direction gradient histogram features, or local features such as SIFT. After quantizing multiple features, a comprehensive score is obtained through linear combination with certain weights as the basis for selecting summaries. After the introduction of deep learning methods, deep features of videos are widely used as picture descriptors. Some existing technologies use low-order features and deep features of videos as input to the HAS-RNN model, and obtain similar experimental results, proving that both low-order features and deep features of pictures have strong descriptive ability [4]. With the development of target recognition models, semantic information of video content is also introduced.

[0006] To generate a video summary, in addition to quantifying the features of individual frames, the features of the summary formed by the combination of multiple frames can also be quantified, such as importance, representativeness, diversity, and plot in some methods [1]. However, due to the large number of frames contained in a video, the number of possible frame combinations is extremely large, and it is generally difficult to calculate all the features of all combinations. In some papers and technologies, the submodularity of importance, representativeness, and diversity is verified [1], which significantly reduces the calculation cost. However, features that do not satisfy submodularity are difficult to be considered, such as the plot mentioned above.

[0007] In addition to calculating various features of a video, another focus of video summarization is how to combine various features. Early researchers regress the weights of various features to establish a linear model to obtain the final score. Recent technologies mainly use deep learning models to train more targeted weights on existing data sets [4]. However, based on the weights obtained by training, although good results can be achieved on the data set used for training, the black-box training is not clear, and the existing data sets are extremely small, and the experimental results are difficult to be convincing. Moreover, if the desired effect is to be achieved, a large amount of model training is required for different video categories and different aesthetic standards. Even if the training is completed, it is difficult to evaluate the pros and cons of the results due to the subjectivity of video summarization [6].

[0008] Currently, deep learning methods still dominate the field of video summarization due to their excellent performance on existing data sets. However, the interpretability of traditional algorithms and the advantages of unsupervised learning make them increasingly valued. The combination of deep learning models and traditional algorithms will be the mainstream trend in the development of video summarization tasks [3, 4, 5, 6].

[0009] The references mentioned in the above text are derived from the following articles:

[0010] [1] Zhao, B., Li, X., Lu, X.: Property-constrained dual learning for video summarization. IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 31(10), pages: 3989-4000, 2019.

[0011] [2] Ma, M., Mei, S., Wan, S., Hou, J., Wang, Z., Feng, D. D.: Video summarization via block sparse dictionary selection. Neurocomputing 378, pages: 197-209, 2020.

[0012] [3] Raj, R., Bhatnagar, V., Singh, A. K., Mane, S., Walde, N.: Video summarization: Study of various techniques. arXiv preprint arXiv:2101.08434, 2021.

[0013] [4] Truong, B. T., Venkatesh, S.: Video abstraction: A systematic review and classification. ACM transactions on multimedia computing, communications, and applications (TOMM) 3(1), pages: 3-40, 2007.

[0014] [5] Tiwari, V., Bhatnagar, C.: A survey of recent work on video summarization: approaches and techniques. Multimedia Tools and Applications 80(18), pages: 27187-27221, 2021.

[0015] [6] K., VV., Sen, D., Raman, B.: Video skimming: Taxonomy and comprehensive survey. ACM Computing Surveys (CSUR) 52(5), pages: 1-38, 2019. SUMMARY

[0016] The present application solves the technical problems:

[0017] The application aims to provide a video summary method based on regret minimization, in particular, a method for obtaining a summary satisfying most people by using regret minimization query technology to intercept part of video pictures.

[0018] The application adopts the following scheme to solve the technical problem:

[0019] A video summary method based on regret minimization comprises the following steps:

[0020] In step one, a video summary candidate frame set is established, and the boundaries of each independent shot in the video are extracted by identifying the video structure. In order to avoid too few candidate frames caused by unclear video structure, candidate frames are extracted again without relying on the video structure. Specifically, after uniformly dividing the video, the multi-dimensional attributes of each frame in each video block are quantized, and the best frame in the block is selected as the summary candidate frame. After obtaining the summary candidate frame set through the two types of methods, the video summary candidate frame set is combined into the final candidate frame set through similarity suppression and certain picture filtering methods. The specific method is as follows:

[0021] Wherein, the judgment of the shot boundary needs to use the cosine similarity and color similarity of the frame and the previous and next fifty frames to calculate the probability of each frame being a shot boundary frame by using TransNet V2. When selecting the best frame in the video block, four indexes of the frame, including stability S, color richness C, representativeness R and clarity E, are extracted as the features of the frame, and then the scores of the four indexes are linearly combined to calculate the total score Score. The score formula is as follows:

[0022] Score(f)=η S ·S(f)+η c ·C(f)+η r ·R(f)+η e ·E(f)

[0023] The weights η S , η c , η r , η e of linear combination are set as 8, 2, 2 and 1 respectively according to the existing research.

[0024] The best frame in the block is selected as a summary candidate frame. After obtaining the summary candidate frame set by two types, the similarity is inhibited by the following formula:

[0025] d i ≤θ d ,d i =D(f i ,f i+1 ),1≤i≤l-1.

[0026] Where d i is the similarity between the i-th frame and the i+1-th frame, θ d is a pre-set similarity threshold, and D(f i , f i+1 ) is the calculation of the similarity between two frames, and the cosine similarity is used in the specific algorithm. After the similarity inhibition, some picture filtering methods are also added, such as removing meaningless pictures with pure color, and finally the boundary frame set and the block representative frame set are merged into the final candidate frame set.

[0027] In step two, the present application combines the regret query method in the method of video summary and multi-criteria decision, and after quantizing the video pictures into multiple criteria, the video summary problem can be converted into a multi-criteria decision query problem. After obtaining the video summary candidate frame set, the feature information of each frame is extracted and quantized, so that each frame can be converted into a multi-dimensional array. The feature information to be processed includes target subject information, motion trajectory information, diversity information, color richness, and image quality in five dimensions, and the score of each dimension is standardized to a score between 0 and 1. Each frame is converted into a 5-dimensional score array. The calculation of the five dimensions covers the low-order features, deep features, semantic features and time sequence features of the picture, and the specific method is as follows:

[0028] The first feature is color richness. A good video summary should be eye-catching, and pictures with rich and bright colors can attract more attention and make people more satisfied. The mainstream method divides the color richness of the picture frame into 1-7 levels of brightness by using the three color channels of the picture. When calculating the picture brightness score, the difference rg between the red channel R and the green channel G is calculated first, and then the difference yb is calculated by the formula , and finally the final score C is obtained. Before that, the overall standard deviation σ rgyb and the average value μ rgyb are calculated according to the standard deviation and the average value of rg and yb, and the formula is as follows:

[0029]

[0030] C=σ rgyb +0.3*μ rgyb

[0031] The second feature is the target subject information. The person and other targets in the picture are identified by the mature Yolov5 framework. Generally, we think that a target occupying a larger area of the picture and closer to the center of the picture is more important, so the invention calculates the area of the target after obtaining the coordinate information of the target in the picture and divides it by the distance from the center of the picture to consider it as the information degree of the target in the picture. The area of the i-th target subject in the picture is denoted as The distance from the center of the picture is denoted as The information degree of the i-th target subject is denoted as

[0032] The information degrees of all targets in the picture are calculated and added to obtain the overall information degree O of the picture.

[0033] The third feature is the motion trajectory information. Generally, a picture containing moving objects contains more information. The invention uses the Shi-Tomasi corner detection algorithm to locate the target objects in the picture, and then uses the pyramid Lucas-Kanade optical flow method to calculate the motion of the objects in the video. The motion information of the target objects is compared with the overall motion information of the video picture. Because the moving speed of the tracked target should be less than the moving speed of the background, the motion of the tracked target in the picture can be obtained by judgment, denoted as the motion trajectory information T.

[0034] The fourth feature is the image quality. A good video frame picture should avoid loss of pixels or distortion of the picture. The invention uses the BRISQUE algorithm to obtain the picture quality score. The BRISQUE algorithm is a non-reference spatial domain image quality evaluation algorithm. It extracts Mean Subtracted Contrast Normalized (MSCN) coefficients from the image, fits the MSCN coefficients into Asymmetric Generalized Gaussian Distribution (AGGD) asymmetric generalized Gaussian distribution, extracts the features of the fitted Gaussian distribution, inputs them into the support vector machine SVM for regression, and obtains the evaluation result of the image quality, denoted as the image quality Q.

[0035] Feature five is diversity information. The frames in the summary should keep diversity as much as possible, but the diversity of the summary frames should be local diversity, i.e. the frame should be quite different from the adjacent frames, but the diversity between the non-adjacent frames should not be too much required, because even if two frames are quite similar, but in different time periods, the information they contain is completely different, for example, similar goal frames in different time periods in a football match video should all be kept in the summary. If a frame has high local diversity, it means that it is very likely to be a completely different shot, and the frame is also likely to be a shot boundary. The invention uses the TransnetV2 model, compares the similarity of the frame and the previous and next 50 frames based on the 3D features of the frame, and obtains the shot boundary prediction value, i.e. the local diversity score D of the frame.

[0036] All the data is normalized to [0, 1] after being obtained by Min-Max standardization, and the higher the value is, the better, and finally a 5-dimensional score of each representative frame is obtained, i.e. each frame in the summary candidate set is converted into the form of [C, O, T, Q, D].

[0037] In step three, the 5-dimensional scores of all candidate frames are compared, and the summary is generated according to the required summary ratio by the regret minimization algorithm, and a summary combination is found which can make people with different preferences in the 5 dimensions all satisfied with the comparison of the summary.

[0038] The regret minimization query proposes the concept of regret rate to measure the satisfaction of users when facing a certain set compared with facing the entire data set, and returns the set that makes the regret rate minimum to the users, which is also called the regret minimization set. Compared with the common Top-k query, the regret minimization query does not need the user to provide a complex utility function to indicate their preferences; compared with the Skyline query, the regret minimization query returns a controllable result set. The most basic idea is to construct a series of representative utility functions, find the points with the maximum utility value on these utility functions, and return these points as the result set.

[0039] Specifically, a utility function f on a d-dimensional point set P can be expressed as f = <f[1], f[2], …, f[d]>, where f[i] ∈ R + , and The specific form of a point in the point set P is p = (p[1], p[2], …, p[d]). The utility value of the point p under the utility function f can be expressed as:

[0040]

[0041] And a utility function f in a point set utility value is the maximum utility value in the point set, called gain, denoted as

[0042] In the present application, the point set is the frame set in the video summary task, and the point set P is denoted as the video frame set V, and the generated summary frame set is denoted as S. The regret of f is the difference between the gain of the small point set and the gain of the large point set, for example, the regret of function f on the summary is r V (S, f) = g (V, f) - g (S, f), and the regret rate is:

[0043]

[0044] The formula can represent a user whose preference is f, and when facing a video V, his regret rate is rr V (S, f). In the present application, we need to consider all user groups, i.e. all possible utility functions f, so we establish a utility function set The discrete distribution in the set has a large number of different utility functions, which can be considered to cover almost all user preferences. On this set representing all users, we can evaluate the maximum regret rate of a video summary The specific formula is as follows:

[0045]

[0046] The formula represents traversing the utility functions of users to calculate the regret rate of each user, selecting the maximum user regret rate as the maximum regret rate of the summary, i.e. the video summary regret rate.

[0047] The task of the present application is to generate a summary with the minimum maximum regret rate. The algorithm used is the most advanced Sphere algorithm, which can efficiently generate points within a subset of no more than a specified number, i.e. generate a video summary of no more than a specified length in the video summary task.

[0048] In step four, the summary result is further supplemented in terms of scenarios to meet the coverage degree in the time sequence of the original video, mainly by observing the time sequence distribution of the frames in the existing summary, and adding appropriate frames between the frames with large time sequence intervals to form the final summary result.

[0049] Observe the time sequence distribution of the frames in the existing summary frame set S, find the maximum time sequence interval, and select the frame with the time sequence located in the middle of the maximum interval from the summary candidate frame set to enter the summary to form a new summary S', and the process is repeated until the number of summary frames meets the requirements. The specific formula is as follows:

[0050] Interval = max(I(f i )-I(f i-1 )), i = 1, 2, …, |S'|-1,

[0051]

[0052] I(f i ) represents the time sequence of the i-th frame, Interval represents the maximum time sequence interval between frames in the existing summary, and λ represents the time sequence coordinate of the new frame to be inserted, that is, adding a new frame into the summary according to the time sequence.

[0053] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0054] (1) The present application makes full use of possible attributes and various advanced technologies to summarize the video, which can better quantify the video picture content and effectively establish a candidate picture set, thereby establishing a better foundation for the generation of the summary.

[0055] (2) The present application innovatively combines the regret minimization algorithm with the video summary task, and uses the regret minimization query to ensure that all users have a certain degree of satisfaction with the summary result without relying on additional information, which has higher universality.

[0056] (3) The present application supplements the time sequence information in the result obtained by the regret minimization through the calculation of the story attribute, so that the present method can meet the summary size while making the effect of the traditional regret minimization algorithm in the video summary task more optimal. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is the overall flowchart of the video summary method based on regret minimization of the present application.

[0058] Figure 2 is the candidate frame screening flowchart of the present application.

[0059] Figure 3 is the candidate frame picture information quantification flowchart of the present application. DETAILED DESCRIPTION

[0060] The technical scheme of the present application will be further described in detail below in combination with the drawings and implementation cases:

[0061] First, we filter the candidate frames of the video, first find the shot boundary frames of the video, for example f1, and then divide the video into uniform blocks, and find the best representative frame in each block by comparing the basic features of each frame in the block, for example f2, and then calculate the similarity between the two frames, if it exceeds the threshold, both frames are kept as summary candidate frames, otherwise one of them is deleted. After comparing all the boundary frames and block representative frames, the summary candidate frame set is obtained.

[0062] Second, the content information of each frame in the summary candidate frame set is generated into its corresponding 5-dimensional score, and the score is standardized between 0 and 1, for example, the score of the summary candidate frame f1 in color richness C, target information O, motion trajectory T, image quality Q, diversity score D is [0.76, 0.87, 0.96, 0.56, 0.88], that is, p = (0.76, 0.87, 0.96, 0.56, 0.88).

[0063] Third, after generating the corresponding 5-dimensional score of each frame in the summary candidate frame set, all the score points are input into the regret minimization query algorithm to filter out the best summary combination, the specific process is as follows, for example, the score of frame f1 is p1 = (0.76, 0.87, 0.96, 0.56, 0.88), and the score of frame f2 is p2 = (0.38, 0.24, 0.15, 0.48, 0.37), frame f2 is much weaker than frame f1 in 5 dimensions, so frame f2 is directly eliminated first, and for frame f3, its 5-dimensional score is p3 = (0.66, 0.77, 0.86, 0.96, 0.78), it can be seen that it is significantly better than frame f1 in the fourth dimension, at this time, the superiority of the two frames needs to be compared, specifically, different users have different preferences for different dimensions, for example, the preference weight of user A for the 5 dimensions is 0.2, 0.2, 0.2, 0.3, 0.1, that is, its utility function f = <0.2, 0.2, 0.2, 0.3, 0.1), then the preference degree of frame f1, that is, the utility value can be considered as f(p1) = 0.2x0.76 + 0.2x0.87 + 0.2x0.96 + 0.3x0.56 + 0.1x0.88 = 0.774, and the preference degree of frame f2 can be considered as f(p2) = 0.2x0.66 + 0.2x0.77 + 0.2x0.86 + 0.3x0.96 + 0.1x0.78 = 0.824, then we will prefer frame f2 as the summary frame, the difference between 0.824 and 0.774 is the regret. When frame f1 and frame f2 are combined into a summary, that is, S = {f1, f2}, the utility value of the summary to user A is the maximum utility value 0.824, if the maximum utility value of user A when facing all the frames V of the entire video is 0.9, then the regret of the summary is r v(S, f) = g(V, f) - g(S, f) = 0.9 - 0.824 = 0.076, the regret rate is When the number of users is large, we cannot meet the best needs of all users, the difference between the results faced by the users and their best choices is the regret, we build a function library that can cover all preference cases, calculate the maximum regret rate that different summary frame combinations may obtain on the basis of it, and select the one with the smallest maximum regret rate as the result output.

[0064] Fourthly, after getting the summary result with the minimum regret, we calculate the time sequence interval between each summary frame, for example, the time sequence of f1, f2, f3 in the result is 530, 630, 1530 respectively, the time sequence between f1 and f2 is the largest, that is, Interval = 1000, then we select the candidate frame closest to the middle of the two frame time sequence numbers from the summary candidate frame set, that is, the time sequence number is 1030, for example, the closest one is found from the candidate frame set, which is the 950th frame, then the 950th frame is added to the summary result, and the process is repeated until the number of frames in the summary result meets the task requirements, for example, the summary result has 4 frames at this time, and the original video is 32 seconds long, generally, one frame of picture can effectively represent the nearby 2 seconds of video original segment, so the summary result can cover 8 seconds of the original video, when the summary task requires a 25% summary ratio, 4 frames of pictures have met the task requirements, that is, the summary result can be output, which is the 530th frame, the 950th frame, the 630th frame and the 1530th frame.

Claims

1. A video summarization method based on regret minimization, characterized by including: The following steps are required: Step 1: Establish a candidate frame set for video summarization. By identifying the video structure, the boundaries of each independent shot in the video are extracted. To avoid insufficient candidate frames due to unclear video structure, candidate frames need to be extracted again without relying on the video structure. After the video is evenly divided into blocks, the multi-dimensional attributes of each frame in each video block are quantified, and the best frame in the block is selected as the candidate frame for summarization. Specifically, the judgment of shot boundaries requires the use of the cosine similarity and color similarity between the frame and the fifty frames before and after it. TransNet V2 is used to calculate the probability that each frame is a shot boundary frame. When selecting the best frame in the video block, four indicators are extracted as the features of the frame: stability (S), color richness (C), representativeness (R), and sharpness (E). Then, the scores of the four indicators are linearly combined to calculate the total score. After obtaining the candidate frame set for video summarization through two methods, the candidate frame set for video summarization is merged into the final candidate frame set through similarity suppression and image filtering. Step 2: Quantize the feature information of each frame in the final candidate frame set so that each frame can be transformed into a multi-dimensional array. The feature information to be processed includes: target subject information, motion trajectory information, diversity information, color richness, and image quality. The score of each dimension is standardized to a number between 0 and 1, and each frame is transformed into a five-dimensional score array. Step 3: Compare the 5-dimensional scores of all candidate frames and generate a summary according to the required summary ratio using the regret minimization algorithm; Step 4: Further supplement the summary results with narrative elements to ensure they cover the time series of the original video. By observing the time series distribution between frames in the existing summary, appropriate frames are added between frames with excessively large intervals to form the final summary result.

2. The video summarization method based on regret minimization as described in claim 1, characterized in that, Various image processing techniques were used in the information quantification process, including: YOLOv5 is used to identify targets, and their subject scores are calculated based on the target's range and distance from the image center. BRISQUE is used to calculate image quality. Cosine similarity is used to calculate image diversity within the video. The Shi-Tomasi corner detection algorithm is used to locate target objects in the image, and the motion of these objects in the video is calculated using the Lucas-Kanade optical flow method. The motion information of the target objects is compared with the overall motion information of the video frame and recorded as motion trajectory information. The difference rg between the red channel R and the green channel G is first calculated, and then combined with the blue channel B using the formula... Calculate the difference yb, and finally, before obtaining the final score C for color richness and vividness, calculate the overall standard deviation σ based on the standard deviation and mean of rg and yb. rgyb and average value μ rgyb The formula is as follows: C=σ rgyb +0.3*m rgyb 。 3. The video summarization method based on regret minimization as described in claim 1, characterized in that: Using the Sphere algorithm to analyze the input multidimensional data, the specific formula for calculating the maximum regret rate of each video summary is as follows: This formula represents the set of functions. The utility function f of the inner traversal of users is used to calculate the regret rate rr of each user's summary S of video V. V (S, f), select the largest user regret rate from them as the maximum regret rate of this summary. This refers to the video summary regret rate.

4. The video summarization method based on regret minimization as described in claim 1, characterized in that, Supplemental selection based on the time sequence of frames in the video: Observe the time series distribution in the existing summary frame set, find the maximum time series interval, and select the frame whose time series is in the middle of the maximum interval from the summary candidate frame set to enter the summary, forming a new summary. Repeat this process until the number of summary frames meets the requirements.

Images