Video expansion method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a video expansion method and device, computer equipment and a storage medium. The method comprises the steps of obtaining a to-be-expanded video clip, performing feature extraction on each frame of video image in the to-be-expanded video clip, and determining feature information in each frame of video image; determining expansion time required to be expanded for the to-be-expanded video clip, and determining a clustering number based on the expansion time; time information corresponding to each frame of video image is determined, the video images are clustered according to the feature information in each frame of video image, the time corresponding to each frame of video image and the clustering number, and a clustering result matched with the clustering number is obtained; and dividing the to-be-expanded video clip into a plurality of video sub-clips according to a clustering result, determining a video expansion mode among the plurality of video sub-clips according to image features of the plurality of video sub-clips, and generating an expanded video clip. By adopting the method, the part needing to be processed can be automatically analyzed, and the display effect of the video obtained after processing is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a data transmission method, device, computer equipment, and storage medium. Background Art

[0002] With the development of various video platforms, users usually use various editing software to edit the videos they have shot. In order to improve the efficiency of video editing and cater to the various preferences of various users, various editing templates are added to the editing software. However, usually, each video clip in each editing template has a fixed duration. If the duration of the video clip put into the editing template by the user is too short, the editing template cannot accurately process the video with too short duration. Therefore, it is necessary to expand the video with too short duration to make it meet the market requirements.

[0003] In traditional expansion technology, some transition effects or slow motion are often added to the beginning and end of videos that are too short to process the videos. However, this method cannot process different types of videos adaptively, and this method can only process the beginning and end, and cannot automatically analyze the part that needs to be processed. The final video effect after processing is not good. Summary of the invention

[0004] Based on this, it is necessary to provide a video expansion method, device, computer equipment, and storage medium to address the above technical problems.

[0005] In a first aspect, the present disclosure provides a video expansion method. The method comprises:

[0006] Acquire a video segment to be expanded, perform feature extraction on each frame of video image in the video segment to be expanded, and determine feature information in each frame of video image;

[0007] Determine an expansion time required for expanding the to-be-expanded video segment, and determine the number of clusters based on the expansion time, wherein the number of clusters is at least greater than 2;

[0008] Determine the time information corresponding to each frame of the video image, cluster the video image according to the feature information in each frame of the video image, the time information corresponding to each frame of the video image and the number of clusters, and obtain a clustering result matching the number of clusters;

[0009] The video segment to be expanded is divided into a plurality of video sub-segments according to the clustering result, and a video expansion method between the plurality of video sub-segments is determined according to image features of the plurality of video sub-segments to generate an expanded video segment.

[0010] In one of the embodiments, dividing the video segment to be expanded into a plurality of video sub-segments according to the clustering result, and determining a video expansion method between the plurality of video sub-segments according to image features of the plurality of video sub-segments to generate an expanded video segment includes:

[0011] Determining cluster boundaries between the clustering results, and segmenting the video segment to be expanded according to the cluster boundaries to obtain a plurality of video sub-segments;

[0012] Determining a video expansion method between the multiple video sub-segments according to the optical flows of the multiple video sub-segments, and processing the multiple video sub-segments according to the determined video expansion method;

[0013] The processed multiple video sub-segments are merged to generate an extended video segment.

[0014] In one embodiment, determining the clustering boundaries between the clustering results includes:

[0015] In response to the existence of a time overlapping region in the clustering results, a cluster boundary between the clustering results is determined based on a time point in the time overlapping region.

[0016] In one of the embodiments, determining the clustering boundaries between the clustering results based on the time points in the time overlapping region includes:

[0017] Determine a time median point in the time overlapping area, and determine a cluster boundary between clustering results according to the time median point;

[0018] Alternatively, determining the total amount of feature information contained in the time overlapping region, and determining the first number of clustering results having the same time overlapping region;

[0019] A time point for equally dividing the total number according to the first number is determined from the time overlapping region, and a clustering boundary between clustering results is determined based on the time point.

[0020] In one embodiment, clustering the video images according to feature information in each frame of the video image, time information corresponding to each frame of the video image, and the number of clusters to obtain a clustering result matching the number of clusters includes:

[0021] Combining the feature information in each frame of the video image and the time information corresponding to each frame of the video image to obtain a cluster feature group;

[0022] Clustering is performed according to the cluster feature group and the cluster quantity to obtain a clustering result.

[0023] In one embodiment, after clustering is performed according to the cluster feature group and the number of clusters to obtain a clustering result, the method further includes:

[0024] Calculating the silhouette coefficient of the clustering result;

[0025] In response to the silhouette coefficient being less than a preset coefficient threshold, decomposing the feature information using a principal component analysis method to obtain feature information of multiple dimensions;

[0026] Combining the feature information of the multiple dimensions with the time information corresponding to each frame of the video image to obtain a modified cluster feature group;

[0027] Clustering is performed according to the modified clustering feature group and the number of clusters to obtain a modified clustering result.

[0028] In a second aspect, the present disclosure further provides a video expansion device. The device comprises:

[0029] A video segment processing module is used to obtain the video segment to be expanded, perform feature extraction on each frame of video image in the video segment to be expanded, and determine feature information in each frame of video image;

[0030] A cluster number determination module, used to determine the expansion time required for the to-be-expanded video segment to be expanded, and determine the number of clusters based on the expansion time, wherein the number of clusters is at least greater than 2;

[0031] A clustering module, used to determine the time information corresponding to each frame of video image, cluster the video image according to the feature information in each frame of video image, the time information corresponding to each frame of video image and the number of clusters, and obtain a clustering result matching the number of clusters;

[0032] The video expansion module is used to divide the video segment to be expanded into multiple video sub-segments according to the clustering result, and determine the video expansion mode between the multiple video sub-segments according to the image features of the multiple video sub-segments to generate an expanded video segment.

[0033] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any of the above method embodiments are implemented.

[0034] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0035] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0036] The video expansion method, device, computer equipment, storage medium and computer program product obtain a video segment to be expanded, extract features from each frame of video image in the video segment to be expanded, and determine the feature information in each frame of video image. By determining the feature information in each frame of video image, subsequent processing and analysis can be facilitated. The expansion time required for the video segment to be expanded is determined, the number of clusters is determined based on the expansion time, the time information corresponding to each frame of video image is determined, and the video image is clustered according to the feature information in each frame of video image, the time information corresponding to each frame of video image and the number of clusters to obtain a clustering result matching the number of clusters. The number of clusters and feature information can be used to accurately cluster video images and classify similar video images into one category. In addition, since the clusters formed by traditional clustering are regular in feature space, they may be disconnected and irregular in time, which will result in the formation of many small video segments (segments clustered into video images). Therefore, in the clustering process, time information can be used for clustering to ensure that the clustering results obtained after clustering are continuous in time. Finally, the video segment to be expanded is divided into multiple video sub-segments according to the clustering result, and the part that needs to be processed can be automatically analyzed. Then, the video expansion method between the multiple video sub-segments is determined according to the image features of the multiple video sub-segments, and an expanded video segment is generated. The video segment can be expanded accurately, and the image features can be used to judge the drastic degree of video changes, so as to select a suitable video expansion method to ensure the effect of video expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of an application environment of a video expansion method in an embodiment;

[0039] Figure 2 is a flow chart of a video expansion method in one embodiment;

[0040] Figure 3 This is a schematic diagram of the process of step S208 in one embodiment;

[0041] Figure 4 is a schematic diagram of a clustering result in one embodiment;

[0042] Figure 5 is a schematic diagram of a time overlap region in one embodiment;

[0043] Figure 6 A schematic diagram of a process for determining cluster boundaries in one embodiment;

[0044] Figure 7 This is a schematic diagram of the process of step S206 in one embodiment;

[0045] Figure 8 is a schematic diagram of a process after step S504 in one embodiment;

[0046] Fig. 9 is a schematic block diagram of the structure of a video expansion device in one embodiment;

[0047] Fig.10 is a schematic diagram of the internal structure of a computer device in one embodiment;

[0048] Fig.11 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0051] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0052] The present disclosure provides a video expansion method, which can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can obtain the video segment to be expanded transmitted from the terminal 102. The server 104 can perform feature extraction on each frame of the video image in the video segment to be expanded, and determine the feature information in each frame of the video image. The terminal 102 or the server 104 can determine the expansion time that the video segment to be expanded needs to be expanded, and determine the number of clusters based on the expansion time, and the number of clusters is at least greater than 2. The server 104 determines the time information corresponding to each frame of the video image, clusters the video image according to the feature information in each frame of the video image, the time information corresponding to each frame of the video image, and the number of clusters, and obtains a clustering result that matches the number of clusters. The server 104 divides the video segment to be expanded into multiple video sub-segments according to the clustering result, and determines the video expansion mode between the multiple video sub-segments according to the image features of the multiple video sub-segments to generate an expanded video segment. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers, panoramic cameras, smart devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers. It should be noted that the above solution can also be implemented using the terminal 102 or the server 104 alone.

[0053] In one embodiment, Figure 2 As shown, a video expansion method is provided, which is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0054] S202, obtaining a video segment to be expanded, performing feature extraction on each frame of video image in the video segment to be expanded, and determining feature information in each frame of video image.

[0055] Among them, in some embodiments of the present disclosure, the video segment to be expanded may be a video segment whose video length needs to be expanded. Feature extraction is an important step in data processing and machine learning. It aims to extract key features or information from raw data that are helpful for model learning. In different fields, the methods and techniques of feature extraction may be different. For example, in the field of image processing, edge detection, color histogram and other methods can be used. Feature information may include: color features, texture features, shape features, etc.

[0056] Specifically, the video segment to be expanded can be obtained, and then each frame of video image contained in the video segment to be expanded can be determined, and features can be extracted from each frame of video image to determine feature information in each frame of video image.

[0057] In some implementation scenarios, when a user needs to edit a video using a video template, the user can input the video clip to be edited into the editing template. Usually, the editing template needs to process clips that meet its duration requirements. For example, the length of the video clip needs to be 5 seconds, and one of the video clips input by the user is 3 seconds, then the 3s video can be the video clip to be expanded.

[0058] S204: Determine an expansion time required for expanding the to-be-expanded video segment, and determine the number of clusters based on the expansion time, wherein the number of clusters is at least greater than 2.

[0059] The number of clusters may generally be the number of cluster centers.

[0060] Specifically, the expansion time required to expand the video segment to be expanded can be calculated according to the duration of the video segment to be expanded and the duration of the video segment required by the video template. After determining the expansion time, the number of clusters can be determined according to the expansion time. Generally, the longer the expansion time, the more clusters there are.

[0061] S206, determining the time information corresponding to each frame of the video image, clustering the video image according to the feature information in each frame of the video image, the time information corresponding to each frame of the video image and the number of clusters, and obtaining a clustering result that matches the number of clusters.

[0062] Specifically, the time information corresponding to each frame of the video image can be determined. Then, the feature information and the time information in each frame of the image are combined. Then, the video image is clustered according to the combined information and the number of clusters to obtain a clustering result that matches the number of clusters.

[0063] In some exemplary embodiments, a clustering algorithm (such as K-means) may be used to cluster the combined information, and the value of K may be set according to the number of clusters during the clustering process.

[0064] S208 , dividing the video segment to be expanded into a plurality of video sub-segments according to the clustering result, and determining a video expansion method between the plurality of video sub-segments according to image features of the plurality of video sub-segments to generate an expanded video segment.

[0065] Among them, image features may include optical flow, pixel value, brightness, etc. Optical flow is an important concept in computer vision and image processing, which refers to the motion information generated by the change in pixel brightness caused by the movement of an object or the movement of a camera in two consecutive frames of images. Optical flow can be used to estimate the speed and direction of the movement of an object in a scene. By analyzing images at different time points, the movement of an object on the image plane can be understood. The video expansion method can usually be a processing method between two video sub-segments, for example, it can include: cyclic alignment, transition effects, slow motion extension, adding static segments, etc. Cyclic alignment is a technology in video expansion and analysis, which is mainly used to process video segments with a cyclic mode, that is, to play the video segments in a loop. Such videos usually contain repeated actions or scenes. Transition effects may include: fade in and fade out, blur gradient, etc. Slow motion extension may be slow motion processing of a video segment. Adding a static segment may be inserting a static picture related to a video sub-segment. In some embodiments of the present disclosure, optical flow is used as an example for illustration.

[0066] Specifically, the video segment to be expanded can be divided into multiple video sub-segments according to the clustering results. Then the optical flow between adjacent video sub-segments is calculated. The drastic degree of picture change between adjacent video sub-segments is determined based on the optical flow. For example, if the optical flow between the two video sub-segments is small, it means that the two videos are both very slow-paced videos, then the video frames can be supplemented between the two video sub-segments by "circular alignment", thereby increasing the video duration. If the optical flow between the two video sub-segments is large, a transition effect can be added between the two video sub-segments, thereby increasing the video duration. It should be noted that those skilled in the art can flexibly select the video expansion method according to the actual situation and the optical flow, and in some embodiments of the present disclosure, there is no restriction on the specific selection of the video expansion method.

[0067] In the above-mentioned video expansion method, a video segment to be expanded is obtained, and feature extraction is performed on each frame of video image in the video segment to be expanded, and feature information in each frame of video image is determined. By determining the feature information in each frame of video image, subsequent processing and analysis can be facilitated. The expansion time required for the video segment to be expanded is determined, the number of clusters is determined based on the expansion time, the time information corresponding to each frame of video image is determined, and the video image is clustered according to the feature information in each frame of video image, the time information corresponding to each frame of video image and the number of clusters, so as to obtain a clustering result matching the number of clusters. The number of clusters and feature information can be used to accurately cluster video images and classify similar video images into one category. In addition, since the clusters formed by traditional clustering are regular in feature space, they may be disconnected and irregular in time, which will result in the formation of many small video segments (segments clustered into video images). Therefore, in the clustering process, time information can be used for clustering to ensure that the clustering results obtained after clustering are continuous in time. Finally, the video segment to be expanded is divided into multiple video sub-segments according to the clustering result, and the part that needs to be processed can be automatically analyzed. Then, the video expansion method between the multiple video sub-segments is determined according to the image features of the multiple video sub-segments, and an expanded video segment is generated. The video segment can be expanded accurately, and the image features can be used to judge the drastic degree of video changes, so as to select a suitable video expansion method to ensure the effect of video expansion.

[0068] In one embodiment, Figure 3 As shown, the method of dividing the video segment to be expanded into a plurality of video sub-segments according to the clustering result, and determining the video expansion mode between the plurality of video sub-segments according to the image features of the plurality of video sub-segments to generate the expanded video segment includes:

[0069] S302: Determine cluster boundaries between the clustering results, and segment the video segment to be expanded according to the cluster boundaries to obtain a plurality of video sub-segments.

[0070] The clustering boundary refers to the boundary or limit used to separate different clusters in cluster analysis. Figure 4 As shown, the clustering results are divided into two categories, namely clustering results A and clustering results B. The cluster boundary in clustering results A may be S1, and the cluster boundary in clustering results B may be S2.

[0071] Specifically, after determining the cluster boundary, the characteristic information contained in the cluster boundary can be determined, the video image corresponding to the characteristic information can be determined, and the determined video image can be divided into one segment, and so on to obtain multiple video sub-segments.

[0072] S304: Determine a video expansion method between the multiple video sub-segments according to the optical flows of the multiple video sub-segments, and process the multiple video sub-segments according to the determined video expansion method.

[0073] S306: Merge the processed multiple video sub-segments to generate an extended video segment.

[0074] Specifically, the optical flows of adjacent frames of video images in adjacent video sub-segments may be calculated. Taking two video sub-segments as an example, the optical flow of the last frame of video images in the previous video sub-segment may be calculated, the optical flow of the first frame of video images in the next video sub-segment may be calculated, and then the optical flow difference between the optical flow of the last frame of video images and the optical flow of the first frame of video images may be calculated, and the video expansion method between the two video sub-segments may be determined according to the optical flow difference. Then, the two video sub-segments are processed according to the determined video expansion method, and then the processed video sub-segments are merged, thereby increasing the duration of the video segment to be expanded.

[0075] In this embodiment, by utilizing cluster boundaries, the video clip to be expanded can be accurately segmented, and similar video images can be divided into one segment to ensure the smoothness of video expansion. In addition, based on the optical flow of the video sub-segments, the degree of picture change between adjacent video segments can also be determined, thereby selecting a suitable video expansion method to further improve the smoothness of the final expanded video segment.

[0076] In one embodiment, determining the cluster boundaries between the cluster results includes:

[0077] In response to the existence of a time overlapping region in the clustering results, a cluster boundary between the clustering results is determined based on a time point in the time overlapping region.

[0078] The time overlapping area can be an area that overlaps in time but belongs to different clustering results. Figure 5 As shown, taking cluster results A and cluster results B as an example, cluster results A and cluster results B overlap in the time period from time X1 to time X2. Therefore, the cluster results in the time period from X1 to X2 can be the time overlapping area.

[0079] Specifically, when there are time overlapping regions in the clustering results, the clustering results need to be divided according to the time points in the time overlapping regions, so as to determine the clustering boundaries between the clustering results.

[0080] Furthermore, Figure 6 As shown, the determining the clustering boundary between the clustering results based on the time points in the time overlapping area includes:

[0081] S402, determining a time median point in the time overlapping area, and determining a clustering boundary between clustering results according to the time median point.

[0082] Specifically, continue with Figure 5 As shown in the figure, the time period corresponding to the time overlapping area ( Figure 5 The time median point is determined by using the time median point to distinguish the cluster results A from the cluster results B. The feature information before the time median point can be classified into the cluster results A, and the feature information after the time median point can be classified into the cluster results B.

[0083] Alternatively, S404, determining the total amount of feature information contained in the time overlapping region, and determining the first number of clustering results having the same time overlapping region.

[0084] S406: Determine a time point for equally dividing the total number according to the first number from the time overlapping region, and determine a clustering boundary between clustering results based on the time point.

[0085] Specifically, the total number of feature information contained in the time overlapping region and the first number of clustering results with the same time overlapping region can be determined. The number of feature information corresponding to each time point corresponding to the time overlapping region can also be determined. The total number is divided equally according to the first number, and then the time point for dividing the total number equally is determined according to the number of feature information corresponding to each time point, and the time point can be a cluster boundary.

[0086] In some exemplary embodiments, for example, the total number of feature information contained in the time overlap region is 120, and the clustering results containing the time overlap region are two. There are five time points in the time overlap region, and the number of feature information corresponding to each time point is 10, 30, 20, 50, and 10, respectively. Therefore, it can be determined that the third time point is the cluster boundary, and there are 60 feature information before the third time point, and there are 60 feature information after the third time point. It can be understood that the above is only for illustration.

[0087] In this embodiment, the clustering results can be accurately divided by the time points in the time overlapping area, thereby ensuring the accuracy of subsequent processing. By using the time median point for division, the time boundary can be quickly determined in the time overlapping area, thereby improving the processing speed. In addition, the total amount of feature information contained in the time overlapping area can be determined; the time point for dividing the total amount in the time overlapping area is determined, and the cluster boundary between the clustering results is determined based on the time point, so that the determined time point can divide the feature information contained in the time overlapping area in half, thereby ensuring that the clustering features on both sides of the cluster boundary in the time overlapping area are basically the same, and ensuring the accuracy of the feature information in each clustering result. There will be no problem of too much or too little feature information due to the uneven distribution of feature information in multiple clustering results due to the appearance of time overlapping areas.

[0088] In one embodiment, Figure 7 As shown, clustering the video images according to the feature information in each frame of the video image, the time information corresponding to each frame of the video image, and the number of clusters to obtain a clustering result matching the number of clusters includes:

[0089] S502: Combine the feature information in each frame of the video image and the time information corresponding to each frame of the video image to obtain a cluster feature group.

[0090] S504: Perform clustering according to the cluster feature group and the number of clusters to obtain a clustering result.

[0091] Specifically, the feature information and the corresponding time in each frame of the video image can be combined to form a new feature group, and the new feature group and the number of clusters are used to perform clustering calculation to obtain the final clustering result.

[0092] In some exemplary embodiments, a roulette wheel algorithm may be used for clustering. For example, a feature point in a feature group may be randomly selected as the first cluster center c1; the shortest distance between each of the remaining feature points in the feature group and the current existing cluster center (i.e., the distance to the nearest cluster center) is calculated, represented by D(x); the larger the value, the greater the probability of being selected as the cluster center; the roulette wheel method is used to select the next cluster center. The above steps are repeated until the same number of cluster centers as the number of clusters are selected.

[0093] In some other exemplary embodiments, the following formula may be used to iteratively adjust the cluster center:

[0094]

[0095] Among them, V represents the total variance or error measure of the clustering, which is usually used to evaluate the tightness of the clustering. The smaller the value, the better the clustering effect. K represents the number of clusters, that is, the number of different clusters into which the data is divided. i Represents the set of all data points in the i-th clustering result. j represents the jth data point belonging to cluster Si. i represents the center (mean) of the i-th cluster, that is, the average value of all data points in cluster Si. tj is the collection time of the j-th data; tistart is the earliest collection time of all data in the i-th class, that is, the start collection time of the class. j ∈[t istart , t end ] means that for any feature point xj in the i-th class Si, its collection time must be in the collection time interval of this class, which means that its collection time is not in other classes, which means that the collection time between the data in each class is continuous.

[0096] In this embodiment, by combining feature information and time information, the image acquisition time corresponding to all feature points in the same class can be made continuous, so that clustering can be performed better.

[0097] In one embodiment, Figure 8 As shown, after clustering is performed according to the cluster feature group and the cluster quantity to obtain a clustering result, the method further includes:

[0098] S602: Calculate the silhouette coefficient of the clustering result.

[0099] S604: In response to the silhouette coefficient being less than a preset coefficient threshold, the feature information is decomposed using a principal component analysis method to obtain feature information of multiple dimensions.

[0100] Among them, the Silhouette Coefficient is an indicator used to evaluate the quality of clustering results, reflecting the similarity between a data point and its cluster, as well as its similarity with other clusters. The value of the Silhouette Coefficient is between -1 and 1, which can help judge the effect of clustering. Principal Component Analysis (PCA) is a commonly used dimensionality reduction technique used to map high-dimensional data to a low-dimensional space while retaining as much original information as possible. Usually, the coefficient threshold can be 0.5.

[0101] Specifically, the silhouette coefficient of each clustering result can be calculated. Then the silhouette coefficient is compared with the preset coefficient threshold. When the silhouette coefficient is less than the coefficient threshold, it can be determined that the clustering quality is poor, indicating that the feature points in the existing feature space are not suitable for clustering into multiple classes. For example, random noise data cannot be clustered into multiple classes. Therefore, for all feature information, assume that their feature vector latitude is n, and perform PCA decomposition on them to obtain new feature information with k latitude.

[0102] S606: Combine the feature information of the multiple dimensions with the time information corresponding to each frame of the video image to obtain a modified cluster feature group.

[0103] S608: Perform clustering according to the modified clustering feature group and the number of clusters to obtain a modified clustering result.

[0104] Specifically, the new feature information and time can be combined to obtain a new cluster feature group, which can be a modified cluster feature group. Then clustering is performed according to the modified cluster feature group and the number of clusters to obtain a modified cluster result. For the specific implementation method here, please refer to the above embodiment, and no repetition is given here. Then, the S208 step mentioned in the above embodiment can be performed here according to the modified cluster result.

[0105] In this embodiment, due to the maximum variance theory of the PCA algorithm and the characteristics of reducing noise interference, the feature vector distance between new feature points is larger than before, and it is easier to find the inherent difference of the data and cluster it into multiple categories, thereby ensuring the accuracy of clustering.

[0106] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiment of the present disclosure also provides a video expansion device for implementing the video expansion method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more video expansion device embodiments provided below can refer to the limitations of the video expansion method above, and will not be repeated here.

[0108] In one embodiment, Fig. 9 As shown, a video expansion device 700 is provided, comprising: a video segment processing module 702, a clustering number determination module 704, a clustering module 706 and a video expansion module 708, wherein:

[0109] The video segment processing module 702 is used to obtain the video segment to be expanded, perform feature extraction on each frame of the video image in the video segment to be expanded, and determine feature information in each frame of the video image;

[0110] A cluster number determination module 704 is used to determine an expansion time required for expanding the video segment to be expanded, and determine the number of clusters based on the expansion time, wherein the number of clusters is at least greater than 2;

[0111] The clustering module 706 is used to determine the time information corresponding to each frame of the video image, cluster the video image according to the feature information in each frame of the video image, the time information corresponding to each frame of the video image and the number of clusters, and obtain a clustering result matching the number of clusters;

[0112] The video expansion module 708 is configured to divide the video segment to be expanded into a plurality of video sub-segments according to the clustering result, and determine a video expansion method between the plurality of video sub-segments according to image features of the plurality of video sub-segments to generate an expanded video segment.

[0113] In one embodiment of the device, the video expansion module 708 includes:

[0114] The video segmentation module is used to determine the clustering boundaries between the clustering results, and segment the video segment to be expanded according to the clustering boundaries to obtain multiple video sub-segments.

[0115] The segmentation processing module is used to determine a video expansion method between the multiple video sub-segments according to the optical flows of the multiple video sub-segments, and process the multiple video sub-segments according to the determined video expansion method.

[0116] The video merging module is used to merge the processed multiple video sub-segments to generate an extended video segment.

[0117] In one embodiment of the device, the video segmentation module further includes: a cluster boundary determination module, which is used to determine the cluster boundaries between the cluster results based on the time points in the time overlapping area in response to the existence of the time overlapping area in the cluster results.

[0118] In one embodiment of the device, the cluster boundary determination module is also used to determine the time median point in the time overlapping area, and determine the cluster boundary between the clustering results based on the time median point; or, determine the total amount of feature information contained in the time overlapping area, and determine the first number of clustering results with the same time overlapping area; determine the time point from the time overlapping area that divides the total amount into two equal parts according to the first number, and determine the cluster boundary between the clustering results based on the time point.

[0119] In one embodiment of the device, the clustering module 706 is further used to combine the feature information in each frame of the video image and the time information corresponding to each frame of the video image to obtain a cluster feature group; clustering is performed according to the cluster feature group and the number of clusters to obtain a clustering result.

[0120] In one embodiment of the device, the device also includes: a dimensionality expansion module, used to calculate the silhouette coefficient of the clustering result; in response to the silhouette coefficient being less than a preset coefficient threshold, using principal component analysis to decompose the feature information to obtain feature information of multiple dimensions.

[0121] The clustering module 706 is further used to combine the feature information of the multiple dimensions with the time information corresponding to each frame of the video image to obtain a modified clustering feature group; and perform clustering according to the modified clustering feature group and the number of clusters to obtain a modified clustering result.

[0122] Each module in the above video expansion device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0123] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as feature information, clustering results, video images, video clips, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video expansion method is implemented.

[0124] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video expansion method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0125] Those skilled in the art will understand that Fig.10 The structure shown in or 11 is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0127] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A video expansion method, characterized in that: The method comprises: Acquire a video segment to be expanded, perform feature extraction on each frame of video image in the video segment to be expanded, and determine feature information in each frame of video image; Determine an expansion time required for expanding the to-be-expanded video segment, and determine the number of clusters based on the expansion time, wherein the number of clusters is at least greater than 2; Determine the time information corresponding to each frame of the video image, cluster the video image according to the feature information in each frame of the video image, the time information corresponding to each frame of the video image and the number of clusters, and obtain a clustering result matching the number of clusters; The video segment to be expanded is divided into a plurality of video sub-segments according to the clustering result, and a video expansion method between the plurality of video sub-segments is determined according to image features of the plurality of video sub-segments to generate an expanded video segment.

2. The method according to claim 1, characterized in that: The step of dividing the video segment to be expanded into a plurality of video sub-segments according to the clustering result, and determining a video expansion method between the plurality of video sub-segments according to image features of the plurality of video sub-segments to generate an expanded video segment includes: Determining cluster boundaries between the clustering results, and segmenting the video segment to be expanded according to the cluster boundaries to obtain a plurality of video sub-segments; Determining a video expansion method between the multiple video sub-segments according to the optical flows of the multiple video sub-segments, and processing the multiple video sub-segments according to the determined video expansion method; The processed multiple video sub-segments are merged to generate an extended video segment.

3. The method according to claim 2, characterized in that The determining the clustering boundaries between the clustering results includes: In response to the existence of a time overlapping region in the clustering results, a cluster boundary between the clustering results is determined based on a time point in the time overlapping region.

4. The method according to claim 3, characterized in that The determining the clustering boundaries between the clustering results based on the time points in the time overlapping area includes: Determine a time median point in the time overlapping area, and determine a cluster boundary between clustering results according to the time median point; Alternatively, determining the total amount of feature information contained in the time overlapping region, and determining the first number of clustering results having the same time overlapping region; A time point for equally dividing the total number according to the first number is determined from the time overlapping region, and a clustering boundary between clustering results is determined based on the time point.

5. The method according to claim 1, characterized in that Clustering the video images according to the feature information in each frame of the video image, the time information corresponding to each frame of the video image, and the number of clusters to obtain a clustering result matching the number of clusters includes: Combining the feature information in each frame of the video image and the time information corresponding to each frame of the video image to obtain a cluster feature group; Clustering is performed according to the cluster feature group and the cluster quantity to obtain a clustering result.

6. The method according to claim 5, characterized in that After clustering is performed according to the cluster feature group and the number of clusters to obtain a clustering result, the method further includes: Calculating the silhouette coefficient of the clustering result; In response to the silhouette coefficient being less than a preset coefficient threshold, decomposing the feature information using a principal component analysis method to obtain feature information of multiple dimensions; Combining the feature information of the multiple dimensions with the time information corresponding to each frame of the video image to obtain a modified cluster feature group; Clustering is performed according to the modified clustering feature group and the number of clusters to obtain a modified clustering result.

7. A video expansion device, characterized in that: The device comprises: A video segment processing module is used to obtain the video segment to be expanded, perform feature extraction on each frame of video image in the video segment to be expanded, and determine feature information in each frame of video image; A cluster number determination module, used to determine the expansion time required for the to-be-expanded video segment to be expanded, and determine the number of clusters based on the expansion time, wherein the number of clusters is at least greater than 2; A clustering module, used to determine the time information corresponding to each frame of video image, cluster the video image according to the feature information in each frame of video image, the time information corresponding to each frame of video image and the number of clusters, and obtain a clustering result matching the number of clusters; The video expansion module is used to divide the video segment to be expanded into multiple video sub-segments according to the clustering result, and determine the video expansion mode between the multiple video sub-segments according to the image features of the multiple video sub-segments to generate an expanded video segment.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images