Video abstract generation method and device, computer device and storage medium

Abstract

The application relates to a video summary generation method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: dividing a target video to be cut into video clips; clustering the video clips according to the similarity between adjacent video clips to obtain a video clip group; calculating a first representative score corresponding to the video clips and a second representative score corresponding to the video clip group to which the video clips belong; selecting a summary video clip from the video clips based on the first representative score and the second representative score, and generating a video summary of the target video according to the summary video clip. The method can ensure that the generated video summary has diversity and representativeness.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video summarization method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, the application and dissemination of video are becoming increasingly widespread, and the amount of video data is also constantly increasing. To facilitate users' quick understanding of video content, video summarization has emerged. Video summarization consists of video frames or segments edited from long videos, containing important information from the original video. How to ensure that video summarization can cover as many actions, scenes, and activities as possible in the original video, and represent the main content of the original video—that is, how to make the generated video summarization diverse and representative—has become an urgent problem to be solved. Summary of the Invention

[0003] Therefore, it is necessary to provide a video summarization method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can generate diverse and representative video summaries, addressing the aforementioned technical problems.

[0004] Firstly, this application provides a video summarization method. The method includes:

[0005] The target video to be edited is divided into video segments;

[0006] Based on the similarity between adjacent video segments, the video segments are clustered to obtain video segment groups;

[0007] Calculate the first representative score corresponding to the video segment, and calculate the second representative score corresponding to the video segment group to which the video segment belongs;

[0008] Based on the first representative score and the second representative score, a summary video segment is selected from the video segments, and a video summary of the target video is generated based on the summary video segment.

[0009] Secondly, this application also provides a video summarization generation apparatus. The apparatus includes:

[0010] The segmentation module is used to divide the target video to be edited into video segments;

[0011] The clustering module is used to cluster the video segments based on the similarity between adjacent video segments to obtain video segment groups;

[0012] The calculation module is used to calculate the first representative score corresponding to the video segment and the second representative score corresponding to the video segment group to which the video segment belongs;

[0013] The generation module is used to select summary video segments from the video segments based on the first representative score and the second representative score, and generate a video summary of the target video based on the summary video segments.

[0014] In one embodiment, the segmentation module is further configured to:

[0015] Frame sampling is performed on the target video to be edited to obtain video sample frames; the target video to be edited is pre-segmented to obtain at least two pre-segmented video segments;

[0016] In the pre-segmented video segments, the segment to which each video sampling frame belongs is determined;

[0017] The target video is re-segmented based on the assigned segment to obtain the target video segment;

[0018] If the number of segmentations does not reach the segmentation threshold, the target video segment is taken as the pre-segmented video segment, and the process returns to the step of determining the segment to which each video sampling frame belongs in the pre-segmented video segment belongs, until the number of segmentations reaches the segmentation threshold.

[0019] The target video segment that reaches the threshold number of segmentations is taken as the video segment obtained by segmenting the target video.

[0020] In one embodiment, the segmentation module is further configured to:

[0021] In the pre-segmented video segments, candidate video segments are determined; the candidate video segments include the current video segment to which the video sampling frame belongs, the previous video segment adjacent to the current video segment, and the next video segment adjacent to the current video segment;

[0022] Calculate the distance between the video sampling frames in the target video and each of the candidate video segments;

[0023] The candidate video segments that meet the distance condition are determined as the segments to which the video sampling frame belongs.

[0024] In one embodiment, the segmentation module is further configured to:

[0025] Obtain the frame image features of each video sampling frame in the target video, and calculate the segment image features of each of the candidate video segments based on the frame image features;

[0026] Based on the sampling time and frame image features corresponding to each video sampling frame, the intermediate time of the pre-segmented video segment, and the segment image features, the distance between each video sampling frame and each candidate video segment is calculated.

[0027] In one embodiment, the segmentation module is further configured to:

[0028] The step of obtaining the frame image features of each video sampling frame in the target video includes:

[0029] Calculate the color histogram of each color channel of the video sampling frame;

[0030] The color histogram is normalized to obtain a normalized histogram;

[0031] The frame image features of the video sampling frame are determined based on the normalized histogram.

[0032] In one embodiment, the clustering module is further configured to:

[0033] Starting from the second video segment in the target video, if the similarity between the current video segment and the adjacent previous video segment is greater than the similarity threshold, the current video segment is assigned to the video segment group to which the previous video segment belongs.

[0034] If the similarity between the current video segment and the previous video segment is less than or equal to the similarity threshold, a new group of video segments is started from the current video segment until the last video segment in the target video.

[0035] In one embodiment, the computing module is further configured to:

[0036] Calculate a first similarity sum; the first similarity sum is the sum of the similarities between the video segment and other video segments in the video segment group to which the video segment belongs;

[0037] Based on the ratio between the sum of the lengths of all video segments in the video segment group to which the video segment belongs and the length of the target video, the weight value corresponding to the video segment group is determined.

[0038] The first representative score is calculated based on the weight value and the first similarity value.

[0039] In one embodiment, the computing module is further configured to:

[0040] Calculate the second similarity sum; the second similarity sum is the sum of the similarities between any video segment in the video segment group to which the video segment belongs and any video segment in other video segment groups outside the video segment group to which the video segment belongs;

[0041] The second representative score is calculated based on the second similarity score, the weight value corresponding to each video segment group, and the number of video segments in each video segment group.

[0042] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0043] The target video to be edited is divided into video segments;

[0044] Based on the similarity between adjacent video segments, the video segments are clustered to obtain video segment groups;

[0045] Calculate the first representative score corresponding to the video segment, and calculate the second representative score corresponding to the video segment group to which the video segment belongs;

[0046] Based on the first representative score and the second representative score, a summary video segment is selected from the video segments, and a video summary of the target video is generated based on the summary video segment.

[0047] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0048] The target video to be edited is divided into video segments;

[0049] Based on the similarity between adjacent video segments, the video segments are clustered to obtain video segment groups;

[0050] Calculate the first representative score corresponding to the video segment, and calculate the second representative score corresponding to the video segment group to which the video segment belongs;

[0051] Based on the first representative score and the second representative score, a summary video segment is selected from the video segments, and a video summary of the target video is generated based on the summary video segment.

[0052] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0053] The target video to be edited is divided into video segments;

[0054] Based on the similarity between adjacent video segments, the video segments are clustered to obtain video segment groups;

[0055] Calculate the first representative score corresponding to the video segment, and calculate the second representative score corresponding to the video segment group to which the video segment belongs;

[0056] Based on the first representative score and the second representative score, a summary video segment is selected from the video segments, and a video summary of the target video is generated based on the summary video segment.

[0057] The aforementioned video summarization generation method, apparatus, computer equipment, storage medium, and computer program product segment the target video to be edited into video segments, and then cluster the video segments according to the similarity between adjacent video segments to obtain video segment groups. Thus, different video content in the target video can be clustered into corresponding video segment groups through segmentation and clustering. The resulting video segment groups can cover diverse actions, scenes, and activities in the target video, ensuring the diversity of the generated video summaries. A first representative score is calculated for each video segment, and a second representative score is calculated for the video segment group to which the video segment belongs. Then, summary video segments are selected from the video segments based on the first and second representative scores, and a video summary of the target video is generated based on these summary video segments. Since the first representative score represents the representativeness of a video segment within its video segment group, and the second representative score represents the representativeness of the video segment group within the entire target video, the summary video segments selected based on the first and second representative scores are representative both within their respective video segment groups and within the entire target video, thus ensuring the representativeness of the generated video summaries. In addition, computer equipment can automatically extract summary video segments from the target video and generate video summaries based on the first representative score and the second representative score, which improves the efficiency of generating video summaries compared to manually selecting summary video segments from the target video. Attached Figure Description

[0058] Figure 1 This is an application environment diagram of the video summarization generation method in one embodiment;

[0059] Figure 2 This is a flowchart illustrating a video summary generation method in one embodiment;

[0060] Figure 3 This is a flowchart illustrating a video segmentation method in one embodiment;

[0061] Figure 4 This is a schematic diagram of a video clip in one embodiment;

[0062] Figure 5 This is a schematic diagram of the segment to which a video sampling frame belongs in one embodiment;

[0063] Figure 6 This is a flowchart illustrating the video segmentation method in another embodiment;

[0064] Figure 7 This is a schematic diagram illustrating the principle of a video segment clustering method in one embodiment;

[0065] Figure 8 This is a flowchart illustrating a method for calculating the first representative score in one embodiment;

[0066] Figure 9 This is a flowchart illustrating a method for calculating a second representative score in one embodiment;

[0067] Figure 10 This is a flowchart illustrating a video summary generation method in another embodiment;

[0068] Figure 11 This is a structural block diagram of a video summarization generation device in one embodiment;

[0069] Figure 12 This is an internal structural diagram of a computer device in one embodiment;

[0070] Figure 13 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0073] The video summary generation method provided in this application can be applied to, for example, Figure 1In the application environment shown, the data storage system stores the data that the computer device 102 needs to process. The data storage system can be integrated onto the computer device 102 or placed in the cloud or on other network servers. The computer device 102 segments the target video to be edited into video segments; based on the similarity between adjacent video segments, it clusters the video segments to obtain video segment groups; it calculates the first representative score corresponding to each video segment and the second representative score corresponding to the video segment group to which the video segment belongs; based on the first and second representative scores, it selects summary video segments from the video segments and generates a video summary of the target video based on the summary video segments. The computer device 102 can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster composed of multiple servers.

[0074] In one embodiment, such as Figure 2 As shown, a video summarization method is provided, which can be applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0075] S202, divide the target video to be edited into video segments.

[0076] The target video can be any type of video, including videos captured by video capture devices, 2D videos captured by panoramic cameras, on-demand videos, cloud-playing videos, animated videos, or game videos. For example, the target video could be a selfie video taken with a handheld device. Another example is a landscape video captured by a camera. Yet another example is a game video transmitted from a cloud gaming server to a game client over the network. The target video can be in various compression formats, including MPEG2 (Moving Picture Experts Group 2), MPEG4 (Moving Picture Experts Group 4), H.264, H.265, and AVI (Audio Video Interleaved) formats. The target video consists of video frames, with a frame rate of 25 frames per second, 30 frames per second, or 24 frames per second. A video clip is a segment composed of multiple video frames. For example, a video clip can be a segment consisting of 125 consecutive frames from the target video, or a continuous segment of the target video lasting 3 seconds.

[0077] In one embodiment, S202 specifically includes: dividing the target video to be edited into equal parts according to its duration to obtain at least two video segments. For example, dividing a 10-minute target video into 10 video segments each with a duration of 1 minute.

[0078] S204. Based on the similarity between adjacent video segments, the video segments are clustered to obtain video segment groups.

[0079] Similarity is used to represent the degree of similarity in content between video clips. For example, if video clip 1 and video clip 2 both show the same scenery, the similarity is 0.9. Or, if video clip 1 and video clip 2 switch scenes to show different scenes, the similarity is 0.2. A video clip group is a collection of one or more video clips; for example, a video clip group could be a collection of video clips from clip 1 to clip 5.

[0080] In one embodiment, before S204, the method further includes: calculating the feature vector of each video segment; and calculating the similarity between adjacent video segments based on the feature vectors of each video segment. The feature vector of a video segment can be calculated based on the feature vectors of each video sampling frame in the video segment. For example, the feature vector of a video segment can be the average of the feature vectors of each video sampling frame in the video segment.

[0081] In one embodiment, the similarity between two video segments is calculated according to formula (1), where S i,j Let i be the similarity between video segment i and video segment j. Let i be the feature vector of video segment i. Let be the feature vector of video segment j.

[0082]

[0083] Computer equipment segments the target video and clusters the video segments based on the similarity between adjacent video clips. Video clips with similar content are grouped together, while those with different content are separated into different groups. Even if the content of a video clip differs significantly from the overall content of the target video, it can still be clustered into an independent group. Video clips in groups of different durations have an equal probability of being selected as video summaries, increasing the probability of shorter video clips being selected and thus ensuring the diversity of the generated video summaries.

[0084] S206, calculate the first representative score corresponding to the video segment, and calculate the second representative score corresponding to the video segment group to which the video segment belongs.

[0085] The first representativeness score indicates the representativeness of a video segment to its group of video segments. After clustering, video segment groups are generally similar in content. The higher the similarity between a video segment and its group, the more representative the video segment is to its group. For example, if a group mainly consists of landscape videos, and one segment is a video of a tourist, that segment has low representativeness and a low first representativeness score. The second representativeness score indicates the representativeness of a video segment group to the target video. The higher the similarity between a video segment group and the target video, the more representative the video segment group is to the target video. For example, if the target video mainly consists of selfies, and a segment group mainly shows the surrounding environment, that segment group has low representativeness and a low second representativeness score; conversely, if a segment group mainly consists of selfies, it has high representativeness and a high second representativeness score.

[0086] S208: Select summary video segments from the video segments based on the first representative score and the second representative score, and generate a video summary of the target video based on the summary video segments.

[0087] The computer device selects the video segments with higher first and second representative scores as the summary video segments. For example, as shown in Table 1, the sum of the first and second representative scores of video segments B and D is higher than 150. The computer device can select video segments B and D as summary video segments and generate a video summary based on video segments B and D.

[0088] Table 1

[0089] Video clip First Representative Score Second Representative Score A 30 70 B 95 80 C 45 67 D 88 70

[0090] In one embodiment, S208 specifically includes: calculating the sum of the first representative score and the second representative score; selecting video segments corresponding to the sum of scores that meet the selection criteria as summary video segments; and generating a video summary of the target video based on the summary video segments. The selection criteria are conditions used to select video segments based on the sum of scores. For example, the selection criteria could be that the sum of scores is greater than a preset threshold; or, for example, the selection criteria could be that the ranking of the sum of scores is within a preset ranking.

[0091] In one embodiment, S208 specifically includes: sorting the first representative score and the second representative score; selecting video segments whose rankings in both the first and second representative score rankings are within a preset range as summary video segments; and generating a video summary of the target video based on the summary video segments. For example, the computer device selects video segments whose rankings in both the first and second representative score rankings are in the top 5 as summary video segments, and generates a video summary of the target video based on the summary video segments.

[0092] In the above embodiments, the target video to be edited is segmented into video segments, and then the video segments are clustered according to the similarity between adjacent video segments to obtain video segment groups. Thus, different video content in the target video can be clustered into corresponding video segment groups through segmentation and clustering. The resulting video segment groups can cover diverse actions, scenes, and activities in the target video, ensuring the diversity of the generated video summary. A first representative score is calculated for each video segment, and a second representative score is calculated for the video segment group to which the video segment belongs. Then, summary video segments are selected from the video segments based on the first and second representative scores, and a video summary of the target video is generated based on these summary video segments. Since the first representative score represents the representativeness of a video segment within its video segment group, and the second representative score represents the representativeness of the video segment group within the entire target video, the summary video segments selected based on the first and second representative scores are representative both within their respective video segment groups and within the entire target video, thus ensuring the representativeness of the generated video summary. In addition, computer equipment can automatically extract summary video segments from the target video and generate video summaries based on the first representative score and the second representative score, which improves the efficiency of generating video summaries compared to manually selecting summary video segments from the target video.

[0093] In one embodiment, such as Figure 3 As shown, S202 specifically includes the following steps:

[0094] S302, perform frame sampling on the target video to be edited to obtain video sample frames; wherein, frame sampling is the sampling of video frames in the target video according to the sampling frequency. The video sample frame is the video frame obtained by frame sampling of the target video. For example, performing frame sampling of P frames per second on a target video with a duration of T seconds yields P×T video sample frames.

[0095] S304, pre-segment the target video to be edited to obtain at least two pre-segmented video segments.

[0096] Pre-segmentation refers to the initial segmentation of the target video. In one embodiment, the target video can be divided equally according to its duration. For example, for a target video with a duration of 5l, the target video can be divided into five pre-segmented video segments of length l, with no overlap between the pre-segmented video segments. For example, as... Figure 4 As shown, the target video is pre-segmented into video segment 1 between frames 1-50, video segment 2 between frames 51-100, and video segment 3 between frames 101-150.

[0097] S306, in the pre-segmented video segment, determine the segment to which each video sampling frame belongs.

[0098] The assigned segment refers to the video segment to which the video sampling frame belongs, including the video segment to which the current sampling frame belongs and the video segments adjacent to it. For example, ... Figure 4 As shown, if video sampling frame 52 belongs to video segment 1, then the segment to which video sampling frame 52 belongs is video segment 1.

[0099] S308, segment the target video based on the attribution segment to obtain the target video segment.

[0100] Computer equipment re-segments the target video based on the assigned segment. The lengths of the resulting target video segments may be the same or different.

[0101] In one embodiment, S308 specifically includes: determining the segment identifier of the segment to which each video sampling frame belongs; if consecutive adjacent video sampling frames have the same segment identifier, then segmenting the consecutive adjacent video sampling frames into the same video segment; if the number of consecutive video sampling frames with the same segment identifier is less than a frame number threshold, then merging them into adjacent video segments to obtain multiple segmented target video segments. For example, such as... Figure 5 As shown, the segment identifier corresponding to video sampling frames 1-3 is 1, and they are segmented into target video segment 1; the segment identifier corresponding to video sampling frames 4-8 is 2, and they are segmented into target video segment 2; the segment identifier corresponding to video sampling frames 9-15 is 3, and they are segmented into target video segment 3.

[0102] S310, if the number of segmentations has not reached the segmentation threshold, the target video segment is used as a pre-segmented video segment, and the process of determining the corresponding segment for each video sampling frame in the pre-segmented video segment is returned until the number of segmentations reaches the segmentation threshold.

[0103] The segmentation threshold can be a preset value, for example, a segmentation threshold of 10. In one embodiment, the segmentation threshold can be determined based on the computing power of the computer device, or it can be determined based on actual needs.

[0104] If the number of segments in a pre-segmented video segment has not reached a threshold after the computer equipment re-segments the segment, it needs to re-segment the segment again until the threshold is reached. Specifically, for example... Figure 6 As shown, after re-segmenting the pre-segmented video segment, the computer device determines whether the number of segmentations has reached the segmentation threshold. If the number of segmentations has not reached the segmentation threshold, the target video segment is used as the pre-segmented video segment, and the pre-segmented video segment is re-segmented again.

[0105] S312, the target video segment that reaches the segmentation threshold is taken as the video segment obtained by segmenting the target video.

[0106] When the threshold for the number of segmentations is reached, the computer device uses the target video segment obtained at this point as the video segment obtained by segmenting the target video.

[0107] In the above embodiments, the target video to be edited is pre-segmented, and the corresponding segment for each video sampling frame is determined in the pre-segmented video segments. The target video is then re-segmented based on the assigned segments until the number of segmentations reaches a segmentation threshold, resulting in video segments obtained from the segmentation of the target video. This allows video content with different characteristics to be segmented into different video segments, thereby enabling the generation of diverse video summaries based on the selected video segments.

[0108] In one embodiment, such as Figure 4 As shown, S304 specifically includes the following steps:

[0109] S402, in the pre-segmented video segments, determine the candidate video segments; the candidate video segments include the current video segment to which the video sampling frame belongs, the previous video segment adjacent to the current video segment, and the next video segment adjacent to the current video segment.

[0110] When segmenting a target video into at least two pre-segmented video segments, for each video sampling frame in the target video, the video sampling frame may belong to its current video segment, or it may belong to the previous or next video segment of the current video segment.

[0111] S404, calculate the distance between the video sampling frames in the target video and each candidate video segment.

[0112] The distance is used to represent the distance between the video sampling frame and each candidate video segment in the time and space dimensions, and can also be used to represent the similarity between the video sampling frame and each candidate video segment.

[0113] In one embodiment, S404 specifically includes: acquiring frame image features of each video sampling frame in the target video, and calculating segment image features of each candidate video segment based on the frame image features; calculating the distance between each video sampling frame and each candidate video segment according to the sampling time corresponding to each video sampling frame and the frame image features, the intermediate time of the pre-segmented video segment and the segment image features.

[0114] The frame image features are the image features of the video sampling frame, which can be color features, contour features, or histogram features of the video sampling frame, etc. In one embodiment, the computer device calculates the color histogram of each color channel of the video sampling frame; normalizes the color histogram to obtain a normalized histogram; and determines the frame image features of the video sampling frame based on the normalized histogram. Specifically, the computer device extracts the three color channels of the RGB format video sampling frame to obtain three grayscale images, and then calculates the color histogram H = {h0, h1, ..., h...} for each grayscale image. N-1}. Where I(x,y) represents the pixel value at coordinates (x,y) in the grayscale image, h i Represents all pixels in a grayscale image The frequency of each grayscale image. After calculating the color histogram of each grayscale image, the computer device normalizes each color histogram. For example, the computer device normalizes the color histogram according to formulas (2) and (3), where, For a normalized histogram, h l This represents the frequency in the color histogram.

[0115]

[0116]

[0117] In one embodiment, the computer device concatenates the normalized histograms of the three color channels of a video sampling frame to obtain the frame image features of the video sampling frame. For each pre-segmented video segment in the target video, the computer device can calculate the average value of the frame image features of each video sampling frame in the pre-segmented video segment, and use this average value as the segment image feature of that pre-segmented video segment. For example, if the target video is segmented into 10 pre-segmented video segments, the computer device will sequentially calculate the segment image features of these 10 pre-segmented video segments.

[0118] In one embodiment, the computer device calculates the distance between the video sampling frame and each candidate video segment according to formula (4). Wherein, dk F is the distance between video sampling frame i and candidate video segment k; i For the segment image features of the current video segment to which the video sampling frame belongs; F k The segment image features of the k-th pre-segmented video segment; t i t represents the sampling time of video sampling frame i; k δ represents the midpoint of the k-th pre-segmented video segment; δ is the distance weight, which can be set according to actual needs.

[0119]

[0120] S406, determine the candidate video segments that meet the distance conditions as the segments to which the video sampling frames belong.

[0121] Among them, the distance condition is the condition for determining whether the candidate video segment is the belonging segment. For example, the distance condition can be the minimum distance, or the distance condition can be less than a preset distance value, etc.

[0122] The computer equipment determines whether the distance between the video sampling frame and each candidate video segment meets the distance condition. If it does, the candidate video segment is determined as the segment to which the video sampling frame belongs. Thus, a video segment that is closest in content to the video sampling frame can be selected from the candidate video segments as the segment to which the video sampling frame belongs.

[0123] In the above embodiments, candidate video segments are determined in the pre-segmented video segments, and the distance between the video sampling frame in the target video and each candidate video segment is calculated. The candidate video segments corresponding to the distances that meet the distance conditions are determined as the segments to which the video sampling frames belong. Thus, the target video can be re-segmented according to the segments to which the segments belong, dividing video content with different content into different video segments, ensuring the diversity of the generated video summaries.

[0124] In one embodiment, S204 specifically includes: starting from the second video segment in the target video, if the similarity between the current video segment and the adjacent previous video segment is greater than a similarity threshold, the current video segment is assigned to the video segment group to which the previous video segment belongs; if the similarity between the current video segment and the previous video segment is less than or equal to the similarity threshold, a new video segment group is started from the current video segment until the last video segment in the target video.

[0125] Specifically, such as Figure 7As shown, if the similarity threshold is set to 0.7, since the similarity between video segment 2 and video segment 1 is 0.78, which is greater than 0.7, video segment 2 is grouped into the same video segment group as video segment 1. Since the similarity between video segment 3 and video segment 2 is 0.4, which is less than 0.7, video segment 3 cannot be grouped into the same video segment group as video segment 2, and a new video segment group is started from video segment 3. Since the similarity between video segment 4 and video segment 3 is 0.8, which is greater than 0.7, video segment 4 is grouped into the same video segment group as video segment 3, and so on, until video segment 8 ends. The eight video segments are divided into three video segment groups: video segments 1 and 2 form video segment group 1, video segments 3-7 form video segment group 2, and video segment 8 forms video segment group 3.

[0126] In the above embodiments, video segments are grouped according to the similarity between adjacent video segments. Video segments with similar content are grouped into the same video segment group, while video segments with dissimilar content are grouped into different video segment groups. This ensures that video segments with significant content differences can be grouped independently, thereby guaranteeing the diversity of the generated video summaries.

[0127] In one embodiment, such as Figure 8 As shown, S206 specifically includes the following steps:

[0128] S802, calculate the first similarity sum; the first similarity sum is the sum of the similarities between the video segment and other video segments in the video segment group to which the video segment belongs.

[0129] Specifically, assuming video segment k i The video clip group it belongs to is g m The computer device calculates the first similarity score according to formula (5). Where S1 is the first similarity score, s i,j For video clip k i With video clip k j Similarity between them.

[0130]

[0131] S804, based on the ratio between the sum of the lengths of all video segments in the video segment group to which the video segment belongs and the length of the target video, determine the weight value corresponding to the video segment group.

[0132] Specifically, the computer device calculates the weight values ​​corresponding to the video segment groups according to formula (6). Where, |k i |For video clips k i Length, The sum of the lengths of all video segments in the video segment group to which the video segment belongs; ∑ j |k j | represents the sum of the lengths of all video segments in the target video, which is also the length of the target video.

[0133]

[0134] S806, the first representative score is calculated based on the weight value and the first similarity value.

[0135] In one embodiment, the computer device calculates the product of the weight value and the first similarity sum to obtain a first representative score. Specifically, the computer device calculates the first representative score rep according to formula (7). local .

[0136]

[0137] In the above embodiments, the computer device calculates a first similarity score between a video segment and other video segments in the same video segment group. Then, based on a weight value determined by the video segment length and the first similarity score, a first representativeness score is calculated. The first representativeness score indicates the representativeness of the video segment within its respective video segment group. Therefore, based on the first representativeness score, video segments with representative content can be selected from the video segment group, ensuring the representativeness of the generated video summary.

[0138] In one embodiment, such as Figure 9 As shown, S206 specifically includes the following steps:

[0139] S902, calculate the second similarity sum; the second similarity sum is the sum of the similarities between any video segment in the video segment group to which the video segment belongs and any video segment in other video segment groups outside the video segment group to which the video segment belongs.

[0140] Specifically, assuming video segment k i The video clip group it belongs to is g m Video clip k j The video clip group it belongs to is g n The computer device calculates the second similarity score according to formula (8). Where S2 is the second similarity score, s i,j For video clip k i With video clip k j Similarity between them.

[0141]

[0142] S904. The second representative score is calculated based on the second similarity score, the weight value corresponding to each video segment group, and the number of video segments in each video segment group.

[0143] In one embodiment, the computer device calculates a second representative score according to formula (9). Wherein, w gn Indicates video clip group g n The corresponding weight value; |g n | indicates a video clip group g n The number of video clips in the data; |g m | indicates a video clip group g m The number of video clips in the video.

[0144]

[0145] In the above embodiments, the computer device calculates a second similarity sum, and based on the second similarity sum, the weight value corresponding to each video segment group, and the number of video segments in each video segment group, calculates a second representative score. The second representative score can represent the representativeness of the video segment group to which the video segment belongs in the target video, ensuring that the selected video segments are representative in the entire target video, thereby ensuring the representativeness of the generated video summary.

[0146] In one embodiment, such as Figure 10 As shown, the video summary generation method includes the following steps:

[0147] S1002, pre-segment the target video to be edited to obtain at least two pre-segmented video segments.

[0148] S1004, in the pre-segmented video segments, determine the candidate video segments; the candidate video segments include the current video segment to which the video sampling frame belongs, the previous video segment adjacent to the current video segment, and the next video segment adjacent to the current video segment.

[0149] S1006, calculate the frame image features of each video sampling frame in the target video, and calculate the segment image features of each pre-segmented video segment based on the frame image features. Calculate the distance between each video sampling frame and each pre-segmented video segment based on the sampling time corresponding to each video sampling frame, the frame image features, the intermediate time of each pre-segmented video segment, and the segment image features.

[0150] S1008: The candidate video segments that meet the distance condition are identified as the assigned segments corresponding to the video sampling frames. The target video is then re-segmented based on the assigned segments to obtain the target video segments.

[0151] S1010, if the number of segmentations has not reached the segmentation threshold, the target video segment is used as a pre-segmented video segment, and the step of determining the corresponding segment of each video sampling frame in the pre-segmented video segment is returned until the number of segmentations reaches the segmentation threshold.

[0152] S1012, the target video segment that reaches the segmentation threshold is taken as the video segment obtained by segmenting the target video.

[0153] S1014, starting from the second video segment in the target video, if the similarity between the current video segment and the adjacent previous video segment is greater than the similarity threshold, the current video segment is assigned to the video segment group to which the previous video segment belongs; if the similarity between the current video segment and the previous video segment is less than or equal to the similarity threshold, a new video segment group is started from the current video segment until the last video segment in the target video.

[0154] S1016, Calculate the first similarity sum; the first similarity sum is the sum of the similarities between the video segment and other video segments in the video segment group to which the video segment belongs.

[0155] S1018, Based on the ratio between the sum of the lengths of all video segments in the video segment group to which the video segment belongs and the length of the target video, determine the weight value corresponding to the video segment group; calculate the first representative score based on the weight value and the first similarity sum value.

[0156] S1020, calculate the second similarity sum; the second similarity sum is the sum of the similarities between any video segment in the video segment group to which the video segment belongs and any video segment in other video segment groups outside the video segment group to which the video segment belongs; calculate the second representative score based on the second similarity sum, the weight value corresponding to each video segment group and the number of video segments in each video segment group.

[0157] S1022, Based on the sum of the first representative score and the second representative score, select summary video segments from the video segments, and generate a video summary of the target video based on the summary video segments.

[0158] The specific content of S1002 to S1022 can be found in the implementation process described above.

[0159] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0160] Based on the same inventive concept, this application also provides a video summarizing apparatus for implementing the video summarizing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more video summarizing apparatus embodiments provided below can be found in the limitations of the video summarizing method described above, and will not be repeated here.

[0161] In one embodiment, such as Figure 11 As shown, a video summarization generation device is provided, including: a segmentation module 1102, a clustering module 1104, a calculation module 1106, and a generation module 1108, wherein:

[0162] The segmentation module 1102 is used to segment the target video to be edited into video segments;

[0163] Clustering module 1104 is used to cluster video segments based on the similarity between adjacent video segments to obtain video segment groups;

[0164] The calculation module 1106 is used to calculate the first representative score corresponding to the video segment and the second representative score corresponding to the video segment group to which the video segment belongs;

[0165] The generation module 1108 is used to select summary video segments from video segments based on the first representative score and the second representative score, and generate a video summary of the target video based on the summary video segments.

[0166] In the above embodiments, the target video to be edited is segmented into video segments, and then the video segments are clustered according to the similarity between adjacent video segments to obtain video segment groups. Thus, different video content in the target video can be clustered into corresponding video segment groups through segmentation and clustering. The resulting video segment groups can cover diverse actions, scenes, and activities in the target video, ensuring the diversity of the generated video summary. A first representative score is calculated for each video segment, and a second representative score is calculated for the video segment group to which the video segment belongs. Then, summary video segments are selected from the video segments based on the first and second representative scores, and a video summary of the target video is generated based on these summary video segments. Since the first representative score represents the representativeness of a video segment within its video segment group, and the second representative score represents the representativeness of the video segment group within the entire target video, the summary video segments selected based on the first and second representative scores are representative both within their respective video segment groups and within the entire target video, thus ensuring the representativeness of the generated video summary. In addition, computer equipment can automatically extract summary video segments from the target video and generate video summaries based on the first representative score and the second representative score, which improves the efficiency of generating video summaries compared to manually selecting summary video segments from the target video.

[0167] In one embodiment, the segmentation module 1102 is further configured to:

[0168] Frame sampling is performed on the target video to be edited to obtain video sample frames; the target video to be edited is pre-segmented to obtain at least two pre-segmented video segments;

[0169] In the pre-segmented video segments, the segment to which each video sampling frame belongs is determined;

[0170] The target video is re-segmented based on the attribution segment to obtain the target video segment;

[0171] If the number of segmentation attempts does not reach the segmentation attempt threshold, the target video segment is used as a pre-segmented video segment, and the process of determining the corresponding segment for each video sampling frame in the pre-segmented video segment is returned until the number of segmentation attempts reaches the segmentation attempt threshold.

[0172] The target video segment that reaches the threshold for the number of segmentations is taken as the video segment obtained by segmenting the target video.

[0173] In one embodiment, the segmentation module 1102 is further configured to:

[0174] In the pre-segmented video segments, candidate video segments are determined; the candidate video segments include the current video segment to which the video sampling frame belongs, the previous video segment adjacent to the current video segment, and the next video segment adjacent to the current video segment.

[0175] Calculate the distance between the video sampling frames in the target video and each candidate video segment;

[0176] The candidate video segments that meet the distance conditions are identified as the segments corresponding to the video sampling frames.

[0177] In one embodiment, the segmentation module 1102 is further configured to:

[0178] Obtain the frame image features of each video sampling frame in the target video, and calculate the segment image features of each candidate video segment based on the frame image features;

[0179] Based on the sampling time and frame image features corresponding to each video sampling frame, the intermediate time of the pre-segmented video segment and the segment image features, the distance between each video sampling frame and each candidate video segment is calculated.

[0180] In one embodiment, the segmentation module 1102 is further configured to:

[0181] Frame sampling is performed on the target video to be edited to obtain video sample frames;

[0182] Obtaining frame image features of each sampled frame in the target video includes:

[0183] Calculate the color histogram of each color channel in the video sample frame;

[0184] Normalize the color histogram to obtain a normalized histogram;

[0185] The frame image features of the video sampling frames are determined based on the normalized histogram.

[0186] In one embodiment, the clustering module 1104 is further configured to:

[0187] Starting from the second video segment in the target video, if the similarity between the current video segment and the adjacent previous video segment is greater than the similarity threshold, the current video segment is classified into the video segment group to which the previous video segment belongs.

[0188] If the similarity between the current video segment and the previous video segment is less than or equal to the similarity threshold, a new group of video segments will start from the current video segment until the last video segment in the target video.

[0189] In one embodiment, the computing module 1106 is further configured to:

[0190] Calculate the first similarity sum; the first similarity sum is the sum of the similarities between the video clip and other video clips in the same video clip group to which the video clip belongs;

[0191] The weight value of the video segment group is determined based on the ratio between the sum of the lengths of all video segments in the video segment group to which the video segment belongs and the length of the target video.

[0192] The first representative score is calculated based on the weight value and the first similarity value.

[0193] In one embodiment, the computing module 1106 is further configured to:

[0194] Calculate the second similarity sum; the second similarity sum is the sum of the similarities between any video segment in the video segment group to which the video segment belongs and any video segment in other video segment groups outside the video segment group to which the video segment belongs;

[0195] The second representative score is calculated based on the second similarity score, the weight value corresponding to each video segment group, and the number of video segments in each video segment group.

[0196] Each module in the aforementioned video summarization generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0197] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores video summarization generation data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a video summarization generation method.

[0198] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a video summarization generation method.

[0199] Those skilled in the art will understand that Figure 12 , 13 The display unit of the computer device shown is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an e-ink display screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad set on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc. The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0200] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0201] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0202] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0204] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video summarization method, characterized in that, The method includes: Frame sampling is performed on the target video to be edited to obtain video sample frames; The target video to be edited is pre-segmented to obtain at least two pre-segmented video segments; In the pre-segmented video segments, the corresponding segment for each video sampling frame is determined; the corresponding segment is the video segment to which the video sampling frame belongs. The target video is re-segmented based on the assigned segment to obtain the target video segment; The target video segment is used as a video segment obtained by segmenting the target video; at least two of the video segments have different video durations; Based on the similarity between adjacent video segments, the video segments are clustered to obtain video segment groups; Calculate the first representative score corresponding to the video segment, and calculate the second representative score corresponding to the video segment group to which the video segment belongs; Based on the first representative score and the second representative score, a summary video segment is selected from the video segments, and a video summary of the target video is generated based on the summary video segment.

2. The method according to claim 1, characterized in that, The method further includes: If the number of segmentations does not reach the segmentation threshold, the target video segment is taken as the pre-segmented video segment, and the process returns to the step of determining the segment to which each video sampling frame belongs in the pre-segmented video segment belongs, until the number of segmentations reaches the segmentation threshold. The target video segment that reaches the threshold number of segmentations is taken as the video segment obtained by segmenting the target video.

3. The method according to claim 1, characterized in that, The step of determining the segment to which each video sampling frame belongs in the pre-segmented video segment includes: In the pre-segmented video segments, candidate video segments are determined; the candidate video segments include the current video segment to which the video sampling frame belongs, the previous video segment adjacent to the current video segment, and the next video segment adjacent to the current video segment; Calculate the distance between the video sampling frames in the target video and each of the candidate video segments; The candidate video segments that meet the distance condition are determined as the segments to which the video sampling frame belongs.

4. The method according to claim 3, characterized in that, The calculation of the distance between the video sampling frame in the target video and each of the candidate video segments includes: Obtain the frame image features of each video sampling frame in the target video, and calculate the segment image features of each of the candidate video segments based on the frame image features; Based on the sampling time and frame image features corresponding to each video sampling frame, the intermediate time of the pre-segmented video segment, and the segment image features, the distance between each video sampling frame and each candidate video segment is calculated.

5. The method according to claim 4, characterized in that, The method further includes: The step of obtaining the frame image features of each video sampling frame in the target video includes: Calculate the color histogram of each color channel of the video sampling frame; The color histogram is normalized to obtain a normalized histogram; The frame image features of the video sampling frame are determined based on the normalized histogram.

6. The method according to claim 1, characterized in that, The step of clustering the video segments based on the similarity between adjacent video segments to obtain video segment groups includes: Starting from the second video segment in the target video, if the similarity between the current video segment and the adjacent previous video segment is greater than the similarity threshold, the current video segment is assigned to the video segment group to which the previous video segment belongs. If the similarity between the current video segment and the previous video segment is less than or equal to the similarity threshold, a new group of video segments is started from the current video segment until the last video segment in the target video.

7. The method according to any one of claims 1 to 6, characterized in that, The calculation of the first representative score corresponding to the video segment includes: Calculate a first similarity sum; the first similarity sum is the sum of the similarities between the video segment and other video segments in the video segment group to which the video segment belongs; Based on the ratio between the sum of the lengths of all video segments in the video segment group to which the video segment belongs and the length of the target video, the weight value corresponding to the video segment group is determined. The first representative score is calculated based on the weight value and the first similarity value.

8. The method according to any one of claims 1 to 6, characterized in that, The calculation of the second representative score corresponding to the video segment group to which the video segment belongs includes: Calculate the second similarity sum; the second similarity sum is the sum of the similarities between any video segment in the video segment group to which the video segment belongs and any video segment in other video segment groups outside the video segment group to which the video segment belongs; The second representative score is calculated based on the second similarity score, the weight value corresponding to each video segment group, and the number of video segments in each video segment group.

9. A video summarization generation device, characterized in that, The device includes: A segmentation module is used to sample frames of the target video to be edited to obtain video sample frames; pre-segment the target video to be edited to obtain at least two pre-segmented video segments; determine the corresponding segment for each video sample frame in the pre-segmented video segments; the corresponding segment is the video segment to which the video sample frame belongs; re-segment the target video based on the corresponding segment to obtain target video segments; and use the target video segments as the video segments obtained by segmenting the target video; at least two video segments have different video durations. The clustering module is used to cluster the video segments based on the similarity between adjacent video segments to obtain video segment groups; The calculation module is used to calculate the first representative score corresponding to the video segment and the second representative score corresponding to the video segment group to which the video segment belongs; The generation module is used to select summary video segments from the video segments based on the first representative score and the second representative score, and generate a video summary of the target video based on the summary video segments.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images