Video cover generation method and device and electronic equipment

By extracting image frames and audio segments from videos and combining audio information to select target image frames, the problem of poor cover quality in existing technologies has been solved, and higher quality video cover generation has been achieved.

CN119031216APending Publication Date: 2024-11-26MICRO DREAM TECHTRONIC NETWORK TECH CHINACO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410948267.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing technologies rely solely on image information when determining video covers, resulting in poor cover quality.

Method used

By extracting image frames and audio segments from the video and combining the audio information to assist in selecting target image frames, the video cover can be determined.

Benefits of technology

We improved the quality of the video covers to ensure they better represent the core content of the video.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031216A_ABST
    Figure CN119031216A_ABST
Patent Text Reader

Abstract

The invention discloses a video cover generation method and device and electronic equipment, and belongs to the field of computers. The method comprises the steps of obtaining a target video; extracting a plurality of image frames from the target video; extracting a plurality of audio segments from the target video; determining at least one target image frame from the plurality of image frames based on the plurality of audio segments; and determining the image frame meeting a target condition in the at least one target image frame as the image frame of the video cover.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of computers, and specifically relates to a method, device and electronic device for generating a video cover. Background Art

[0002] With the rapid development of the Internet, video-related applications are emerging in an endless stream. In the process of video dissemination and sharing, a sufficiently attractive video cover can bring higher number of viewers and playback volume. Therefore, how to obtain an attractive video cover is very important.

[0003] Related technologies usually analyze and process the image information in the video to determine the video cover of the video. The video cover determined in this way has the problem of poor quality. Summary of the invention

[0004] The embodiments of the present application provide a method, device and electronic device for generating a video cover, which can solve the problem of poor quality of video covers determined in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a video cover, comprising:

[0006] Get the target video;

[0007] Extracting multiple image frames from the target video;

[0008] extracting a plurality of audio segments from the target video;

[0009] Based on the plurality of audio segments, determining at least one target image frame from the plurality of image frames;

[0010] An image frame that meets the target condition among the at least one target image frame is determined as an image frame of the video cover.

[0011] In a second aspect, an embodiment of the present application provides a device for generating a video cover, including:

[0012] An acquisition module, used to acquire a target video;

[0013] An extraction module, used to extract multiple image frames from the target video; extract multiple audio segments from the target video;

[0014] A determination module is used to determine at least one target image frame from the multiple image frames based on the multiple audio segments; and determine an image frame that meets a target condition among the at least one target image frame as an image frame of a video cover.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores a program or instruction running on the processor, and when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method described in the first aspect.

[0018] At least one of the above technical solutions provided in the embodiments of the present application can achieve the following technical effects:

[0019] In an embodiment of the present application, a target video is obtained; multiple image frames are extracted from the target video; multiple audio segments are extracted from the target video; based on the multiple audio segments, at least one target image frame is obtained from the multiple image frames; and an image frame that meets the target condition in the at least one target image frame is determined as an image frame of the video cover. In this way, in the process of determining the video cover, not only the image information of the video is considered, but also the audio information of the video is considered, and at least one audio segment is used to assist in determining the at least one target image frame. Compared with the related art that only considers the image information to determine the video cover, the determined video cover has higher quality, which solves the problem of poor quality of the video cover determined in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flow chart of a method for generating a video cover provided in an embodiment of the present application;

[0022] Figure 2 is a flow chart of a method for generating a video cover provided in an embodiment of the present application;

[0023] Figure 3 is a flow chart of a method for generating a video cover provided in an embodiment of the present application;

[0024] Figure 4 This is a general concept diagram of a method for generating a video cover provided in an embodiment of the present application;

[0025] Figure 5 This is a complete flow chart of a method for generating a video cover provided in an embodiment of the present application;

[0026] Figure 6 It is a structural block diagram of a device for generating a video cover provided in an embodiment of the present application;

[0027] Figure 7 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The method for generating a video cover provided in the embodiment of the present application is applied to the scene of video processing, and in particular, can be applied to the generation of a video cover. Specifically, the video cover of the target video can be generated through the image information and audio information of the target video.

[0032] The method for generating a video cover provided in an embodiment of the present application can be executed by a target device, wherein the target device can be one electronic device or multiple electronic devices. That is to say, the method for generating a video cover provided in an embodiment of the present application can be executed by an electronic device, wherein the electronic device can be, for example, a terminal device such as a desktop computer, a laptop computer, a mobile phone, a tablet, or a server, such as an independent physical server, a server cluster composed of multiple servers, and a cloud server capable of cloud computing. In the case where the method for generating a video cover provided in an embodiment of the present application is executed by multiple electronic devices, these multiple electronic devices can form a service cluster, which cooperate with each other to complete each step.

[0033] The following is a detailed description of the method for generating a video cover provided in the embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0034] See also Figure 1 , Figure 1 The following is a flow chart of a method for generating a video cover provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0035] Step 110: Obtain target video;

[0036] In an embodiment of the present application, the target video is an original video used to generate a video cover. The target video can be uploaded by a user or downloaded via the Internet. At the same time, there are no restrictions on the video length and video content of the target video. That is, the target video can be a short video (for example, less than 3 minutes), a medium-long video (for example, 3 minutes to 20 minutes), or a long video (for example, more than 20 minutes). And depending on the content of the video, the target video can be, for example, a game video, a life video, a science video, and a music video.

[0037] Step 120: extracting multiple image frames from the target video;

[0038] In the embodiment of the present application, the image frames in the target video are extracted by a video frame extraction tool (such as FFmpeg or other tools) to obtain multiple image frames. There is no requirement for the extraction interval of the image frames, and multiple image frames can be randomly selected from the image frames of the video as image frames; the image frames can also be extracted at a certain time interval (such as 1 second), for example, if the length of the video is 30 seconds, frame extraction is performed every one second to obtain 30 image frames.

[0039] Step 130: extracting multiple audio segments from the target video;

[0040] In an embodiment of the present application, the audio segments in the target video are extracted by an audio extraction tool (e.g., AVlto tool) to obtain multiple audio segments. The multiple audio segments may be, for example, multiple non-overlapping audio segments selected from the target video, or multiple overlapping audio segments selected from the target video.

[0041] Step 140: determining at least one target image frame from the plurality of image frames based on the plurality of audio segments;

[0042] In an embodiment of the present application, the target image frame may be an image frame that can better convey the core content of the target video among multiple image frames. In order to fully consider the core content of the target video, at least one target image frame that can better convey the core content of the target video may be selected from multiple image frames by analyzing the audio information and image information of the target video. The multiple audio segments are used to assist in determining the at least one target image frame. Specifically, the audio information is used to assist in determining at least one target image frame from multiple image frames.

[0043] In an embodiment of the present application, it is possible to further extract the text content corresponding to the multiple audio segments. By analyzing the text content of the multiple audio segments, it can be considered that the recurring text content can convey the core content of the target video to a certain extent. Based on this, at least one target image frame can be determined from multiple image frames based on the recurring text content. For example, if apples appear repeatedly in the multiple audio segments, an image frame in which apples appear can be selected from the multiple image frames and determined as the target image frame.

[0044] In one embodiment of the present application, the number of the multiple image frames is N, the number of the multiple audio segments is N, and the N image frames correspond one-to-one to the N audio segments; the extracted image frames and audio segments have indexes numbered according to the same rule, and N is a positive integer; the at least one target image frame is determined from the clustering results corresponding to the N image frames based on the index of the target audio segment in the N audio segments, and the target audio segment is selected from the clustering results corresponding to the N audio segments.

[0045] In the embodiment of the present application, the number of image frames and the number of audio segments extracted from the target video are both N, and there is a one-to-one correspondence between the N image frames and the N audio segments. Specifically, the time period when the audio segment is mapped to the target video may include the time point when the image frame corresponding to the audio segment is mapped to the target video.

[0046] It should be noted that there is no restriction on the time length of the audio segment, and the time length of the audio segment can be consistent with the extraction time interval of the image frame. For example, if the length of the video is 30 seconds, the audio of the video is extracted according to the duration of 1 second to obtain 30 audio segments, wherein the audio segment contents of the 30 audio segments are not repeated. The time length of the audio segment can be inconsistent with the extraction time interval of the image frame. For example, if the time length of the audio segment is 4 seconds, and the extraction time interval of the image frame is 1 second. Specifically, if the length of the video is 30 seconds, a 4-second audio segment can be extracted every 1 second, and the audio segment of the 0th to 4th seconds, the audio segment of the 1st to 5th seconds, and the audio segment of the 2nd to 6th seconds can be obtained, and so on. 30 audio segments of 4 seconds in length can be obtained, wherein the audio segment contents of the 30 audio segments are repeated.

[0047] In addition, the N image frames and the N audio segments have indexes numbered according to the same rule, where N is a positive integer. Specifically, the N image frames and the N audio segments may be indexed according to the same rule, and the rule may be indexed according to the time sequence of the time points mapped to the target video, such as 0, 1, 2, 3, etc. Since the N image frames and the N audio segments correspond one to one and have indexes numbered in sequence according to the same rule, the image frames and audio segments that have a corresponding relationship may have the same index. For example, if there is a corresponding relationship between image frame A and audio segment A, then image frame A and audio segment A have the same index.

[0048] In an embodiment of the present application, N audio segments may be clustered first, and a target audio segment may be selected based on the clustering results of the N audio segments. After determining the target audio segment, the index of the target audio segment may be determined. Then, the N image frames may be clustered to obtain clustering results of the N image frames, and the clustering results may include multiple, and each clustering result may include one or more image frames. When the number of image frames included in the clustering result is more, the image frames in the clustering result can better convey the content of the target video, and a target clustering result may be selected from multiple clustering results. The target clustering result may be one or more clustering results in which the number of image frames is greater than the image frame number threshold, or the target clustering result may be one or more clustering results in which the number of image frames is larger. Then, at least one target image frame may be determined from the N image frames based on the index of the target audio segment.

[0049] Specifically, in one embodiment of the present application, the step 140 of obtaining at least one target image frame from the multiple image frames based on the multiple audio segments includes: clustering the N image frames to obtain K first clustering results; clustering the N audio segments to obtain S second clustering results; determining P first clustering results that meet the first condition from the K first clustering results; determining Q second clustering results that meet the second condition from the S second clustering results; based on the Q second clustering results, determining at least one target image frame from the P first clustering results, the index of the target image frame being in the index of the target audio segment included in the Q second clustering results; wherein K, S, P and Q are all positive integers, P is less than or equal to K, and Q is less than or equal to S.

[0050] Step 150: Determine an image frame that meets the target condition among the at least one target image frame as an image frame of the video cover.

[0051] In an embodiment of the present application, the target condition may include a user's cover requirement for a target video cover, and may also include a cover requirement determined according to an application scenario of the video. It is further determined whether each of the at least one target image frame meets the target condition, and the target image frame that meets the condition is determined as an image frame of the video cover.

[0052] In one embodiment of the present application, the target condition includes: at least one of a contrast condition, a brightness condition, a clarity condition and an illumination condition. The step 150 of determining the image frame that meets the target condition in the at least one target image frame as the image frame of the video cover includes: determining A target image frames that meet the target condition from the at least one target image frame; wherein A is a natural number; when A is equal to B, determining the A target image frames as the image frames of the video cover; when A is greater than B, obtaining a weight score for each of the A target image frames; obtaining B target image frames with the highest weight scores from the A target image frames as the image frames of the video cover; when A is less than B, reacquiring at least one target image frame; determining the image frame that meets the target condition in the at least one target image frame that is reacquired as the image frame of the video cover; wherein B is a preset number of covers

[0053] In an embodiment of the present application, a target video is obtained; multiple image frames are extracted from the target video; multiple audio segments are extracted from the target video; at least one target image frame is determined from the multiple image frames based on the multiple audio segments; and an image frame that meets the target condition in the at least one target image frame is determined as an image frame of the video cover. In this way, in the process of determining the video cover, not only the image information of the video is considered, but also the audio information of the video is considered, and at least one audio segment is used to assist in determining the at least one target image frame. Compared with the related art that only considers the image information to determine the video cover, the determined video cover has higher quality, which solves the problem of poor quality of the video cover determined in the related art.

[0054] See also Figure 2 , Figure 2 A flowchart of a method for generating a video cover provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0055] Step 210: Acquire the target video;

[0056] Step 220: extracting N image frames from the target video;

[0057] Step 230: extracting N audio segments from the target video, wherein the N image frames correspond to the N audio segments one by one; the extracted image frames and audio segments have indexes numbered according to the same rule, and N is a positive integer;

[0058] Step 240: performing clustering processing on the N image frames to obtain K first clustering results;

[0059] In an embodiment of the present application, in order to determine an image frame that can better convey the core content of the target video from N image frames, the N image frames can be clustered. By clustering the N image frames, K first clustering results can be obtained. Through the K first clustering results, similar image frames are classified into one first clustering result, and then by analyzing the clustering results, similar image frames are analyzed. In an embodiment of the present application, there is no limitation on the specific clustering algorithm, and the clustering algorithm can be, for example, a K-means clustering algorithm and a hierarchical clustering algorithm.

[0060] Exemplarily, in one embodiment of the present application, in step 240, clustering is performed on the N image frames to obtain K first clustering results, including: performing image feature extraction on the N image frames to obtain a first feature map; based on the first feature map, calculating the image similarity between any two image frames in the N image frames; based on the image similarity, clustering is performed on the N image frames to obtain K first clustering results; the image similarity between any two image frames in the first clustering result is greater than a set first threshold.

[0061] In an embodiment of the present application, feature extraction can be performed through a deep learning model, and the deep learning model may include at least one of a Visual Geometry Group (VGG) network model, a Residual Neural Network (ResNet) model, and an inception network model. Through the deep learning model, feature extraction can be performed on each of the N image frames. For example, feature extraction is performed on each of the 30 image frames, and 30 corresponding feature vectors can be obtained to obtain a first feature map. Through the deep learning model, a powerful feature extraction capability can be obtained, and high-dimensional feature expressions can be learned from massive data, so that the generated dynamic cover can more accurately express the original video content.

[0062] In the embodiment of the present application, after feature extraction is performed on N image frames, the similarity between any two image frames in the N image frames can be calculated according to the first feature map to obtain a similarity matrix of the N image frames. Through the similarity matrix, the similarity relationship between any two image frames in the N image frames can be obtained.

[0063] In the embodiment of the present application, in order to further quantify the judgment condition for determining whether two image frames are similar, a first threshold value (for example, 0.8) may be set. If the similarity between the two image frames is greater than the first threshold value, the two image frames may be considered similar. In addition, there is no restriction on the setting method of the first threshold value, that is, the first threshold value may be fixedly set to 0.8, and the first threshold value may also be a dynamic threshold value, for example, multiple values ​​may be randomly selected within a certain range (0.7 to 0.9), and then averaged, and the averaged result is used as the first threshold value.

[0064] In the embodiment of the present application, when determining the similarity between any two image frames in the N image frames, similar image frames can be clustered, and finally K first clustering results can be obtained. Each of the K first clustering results is similar to each other, and any two of the K first clustering results are not similar to each other.

[0065] In an embodiment of the present application, N image frames can be clustered according to the similarity between each other to obtain K first clustering results. The first clustering results can be used to further determine the image frame of the video cover from the N image frames, thereby improving the accuracy of selecting the image frame of the video cover.

[0066] Step 250: performing clustering processing on the N audio segments to obtain S second clustering results;

[0067] In the embodiment of the present application, S second clustering results can be obtained by clustering N audio segments. Through the S second clustering results, similar audio segments are classified into one second clustering result, and then the analysis of the clustering results is performed to achieve the analysis of similar audio segments. In the embodiment of the present application, there is no limitation on the specific clustering algorithm, and the clustering algorithm can be, for example, a K-means clustering algorithm and a hierarchical clustering algorithm.

[0068] Exemplarily, in one embodiment of the present application, clustering is performed on the N audio segments in step 250 to obtain S second clustering results, including: extracting audio features from the N audio segments to obtain a second feature map; based on the second feature map, calculating the audio similarity between any two audio segments in the N audio segments; based on the audio similarity, clustering is performed on the N audio segments to obtain S second clustering results; the audio similarity between any two audio segments in the second clustering results is greater than a set second threshold.

[0069] In an embodiment of the present application, before audio features are extracted from the N audio segments, the N audio segments may be preprocessed, and the preprocessing may include at least one of channel conversion, pre-emphasis, resampling, framing, and windowing. Features of the N audio segments may be extracted using a deep learning model. The deep learning model used for feature extraction of the N audio segments and the deep learning model used for feature extraction of the N image frames may be one deep learning model.

[0070] In the embodiment of the present application, due to the obvious serialization characteristics of the audio itself, if the duration of the audio segment is consistent with the selection interval of the image frame, for example, the duration of the audio segment is 1 second. At this time, the audio feature extraction of 1 second of audio, and the similarity calculation through the extracted feature vector, will result in low distinction between the audio segments, making it difficult to judge the audio quality. Therefore, if the selection interval of the image frame is short, the step size of the audio segment can be set, and the audio segment of the step size can be selected. The step size of the audio segment can be greater than the selection interval of the image frame. For example, the selection interval of the image frame is 1 second, and the step size of the audio segment can be selected as 4 seconds. The N audio segments can be selected from the 0th to 4th seconds, the 1st to the 5th seconds, etc. of the target video.

[0071] In an embodiment of the present application, audio features are extracted for each of the N audio segments by adopting a step size, and the vectors obtained by the audio feature extraction can be averaged by the step size to obtain a second feature map. Then, the audio similarity between any two audio segments in the N audio segments is calculated based on the second feature map, and the N audio segments are clustered based on the audio similarity. Among them, the audio similarity between the two audio segments may include the similarity of the audio content corresponding to the two audio segments, such as a certain keyword appearing in both audio segments. If the audio similarity between the two audio segments is greater than the set second threshold, it can be considered that the two audio segments are in the same clustering result. The process of calculating the similarity and clustering the N audio segments here can refer to the process of calculating the similarity and clustering the N image frames in step 240.

[0072] In an embodiment of the present application, N audio segments can be clustered by comparing their similarities to obtain S second clustering results. The second clustering results can be used to determine the image frame of the video cover, thereby improving the accuracy of selecting the image frame of the video cover.

[0073] Step 260: Determine P first clustering results that meet a first condition from the K first clustering results;

[0074] In an embodiment of the present application, the K first clustering results can be further screened by a first condition, and the first condition can include at least one of a large number of image frames included in the first clustering result and a relatively scattered time points corresponding to the image frames included in the first clustering result. For example, due to the continuity of the target video, image frames with adjacent time points are likely to be similar, and if the time points corresponding to the image frames included in a certain first clustering result are relatively scattered, it means that similar image frames in the first clustering result appear repeatedly in the entire video, and it can be considered that the image frames in the clustering result can more fully convey the core content of the target video.

[0075] Exemplarily, in one embodiment of the present application, the step 260 of determining P first clustering results that meet the first condition from the K first clustering results includes: determining the number of image frames in each of the K first clustering results; sorting the K first clustering results from most to least according to the number of image frames to obtain a first sorting result; and obtaining the top P first clustering results from the first sorting result.

[0076] In an embodiment of the present application, if the number of image frames in a first clustering result is large, it can be considered that the proportion of relevant information appearing in the target video is large, which means that the target video is more inclined to express relevant information. The image frames in the first clustering result can be used as alternative image frames for the video cover.

[0077] In the embodiment of the present application, the number of image frames in each of the K first clustering results can be counted, and the K first clustering results can be sorted according to the statistical results to obtain the first sorting result, and the first P first clustering results with the largest number of image frames in the first sorting result can be used as the P first clustering results, and then the candidate image frames for the video cover can be further determined from the P first clustering results. In this way, the proportion of relevant information can be more objectively judged through the number of image frames in the clustering results, and the accuracy of selecting the image frames for the video cover can be improved.

[0078] Step 270: Determine Q second clustering results that meet a second condition from the S second clustering results;

[0079] In an embodiment of the present application, the S second clustering results can be further screened by a second condition, and the second condition may include at least one of a large number of audio segments included in the second clustering results and a relatively dispersed time points corresponding to the audio segments included in the first clustering results.

[0080] In an embodiment of the present application, if the audio segment is selected using a step size method, before determining the index, it is necessary to reverse-parse the N audio segments according to the step size. For example, if the step size is 4 seconds, the index of the audio segment corresponding to the 1st to 5th seconds can be determined, and the index corresponds to the image frame of the 1st second.

[0081] Exemplarily, in one embodiment of the present application, the step 270 of determining Q second clustering results that meet the second condition from the S second clustering results includes: determining the number of audio segments in each of the S second clustering results; sorting the K first clustering results from most to least according to the number of audio segments to obtain a second sorting result; and obtaining the top Q second clustering results from the second sorting result.

[0082] In an embodiment of the present application, if the number of audio segments in a second clustering result is large, it can be considered that the proportion of relevant information appearing in the target video is large, which means that the target video is more inclined to express relevant information. The audio segment in the second clustering result can be used as an alternative image frame for the video cover.

[0083] In an embodiment of the present application, the number of audio segments in each of the S second clustering results can be counted, and the S second clustering results can be sorted according to the statistical results to obtain a second sorting result, and the first Q second clustering results with the largest number of audio segments in the second sorting result can be used as Q second clustering results, and then the candidate image frames of the video cover can be determined through the Q second clustering results. In this way, the proportion of relevant information can be more objectively judged through the number of audio segments in the clustering results, and the accuracy of selecting the image frame of the video cover can be improved.

[0084] Step 280: Based on the Q second clustering results, determine at least one target image frame from the P first clustering results, where the index of the target image frame is in the index of the target audio segment included in the Q second clustering results;

[0085] Among them, K, S, P and Q are all positive integers, P is less than or equal to K, and Q is less than or equal to S.

[0086] In an embodiment of the present application, a recurring image frame can be selected from the Q second clustering results and the P first clustering results, and then the image frame can be determined as a target image frame. The index of the image frame can be used to indicate the corresponding time point of the image frame in the target video, and the index of the audio segment is used to indicate the corresponding time period of the audio segment in the target video. Specifically, the target image frame has a corresponding index in the P first clustering results, and the index of the target image frame has the same index in the Q second clustering results.

[0087] For example, 30 image frames and 30 audio segments can be extracted, wherein the 30 image frames and the 30 audio segments can correspond one to one. According to the time point sequence of the target video, the corresponding indexes of the 30 image frames can be 1, 2, 3, ... 30 respectively; the corresponding indexes of the 30 audio segments can be 1, 2, 3, ... 30 respectively. At this time, the image frame with an index of 1 corresponds to the audio segment with an index of 1. After clustering the 30 image frames and the 30 audio segments, the P first clustering results obtained include image frames with indices of 3, 4, 7, and 9, and the Q second clustering results obtained include target audio segments with indices of 4, 5, 9, and 12, then it can be determined that the indexes of the target image frames are 4 and 9.

[0088] Exemplarily, in one embodiment of the present application, the step 280 of determining at least one target image frame from the P first clustering results based on the Q second clustering results includes: for each of the P first clustering results, performing the following operations: determining respectively whether the index of each image frame in the current first clustering result is the same as the index of any target audio segment in the Q second clustering results; if so, determining that the current image frame is the target image frame.

[0089] In the embodiment of the present application, it is possible to determine whether an image frame in each of the P first clustering results has a corresponding audio segment in the Q second clustering results. The judgment can be made by index. Specifically, for each of the P first clustering results, it is possible to determine whether each index in the clustering result has the same index in the Q second clustering results. If there is an identical index, the image frame corresponding to the index can be determined as the target image frame.

[0090] In the embodiment of the present application, the target image frame is determined by indexing, which can achieve more efficient and rapid determination of the target image frame and has better accuracy.

[0091] In one embodiment of the present application, after determining at least one target image frame from the P first clustering results based on the Q second clustering results in step 280, the method further includes: when the number of the target image frames is less than the preset cover number, increasing the value of P.

[0092] In an embodiment of the present application, considering that there may be a requirement for a preset number of covers for the video cover, for example, requiring the video cover to contain 20 images, it is possible to first determine whether M is not less than the preset number of covers, that is, determine whether M is not less than 20. In the case where M is not less than the preset number of covers, the image frame of the video cover can be further determined from the M target image frames. In the case where M is less than the preset number of covers, the value of P can be increased, and the process can be returned to step 260 to determine the P first clustering results again and perform subsequent steps. When increasing the value of P, P can be increased by a certain proportion (for example, 40%).

[0093] Correspondingly, Q can be adjusted with reference to the adjustment of the value of P, for example, by increasing the value of Q, the process can return to step 270, determine Q second clustering results again, and continue with subsequent steps. Specifically, when the value of P is increased, the value of Q can be increased at the same time; or, when the value of P is increased, the value of Q can remain unchanged.

[0094] In the embodiment of the present application, if the determined target image frames are still less than the preset cover quantity after the value of P is adjusted once, the value of P can be adjusted a second time, that is, the value of P can be increased again. When the determined target image frames do not meet the requirement of the preset cover quantity, the value of P can be adjusted multiple times until the number of determined target image frames meets the requirement of the preset cover quantity, and the process of adjusting the value of P can be stopped.

[0095] In an embodiment of the present application, when the number of target image frames is less than the preset number of covers, the screening range of the target image frames can be enlarged by increasing the value of P, and more target image frames can be further determined, thereby ensuring that the number of target image frames determined meets the requirements of the preset number of covers and better meets user needs.

[0096] Step 290: Determine an image frame that meets the target condition among the at least one target image frame as an image frame of the video cover.

[0097] In the embodiment of the present application, clustering processing is performed to determine the image frames and audio segments that meet the conditions from N image frames and N audio segments, and then determine the target image frame. In this way, the target image frame whose image information and audio information meet the conditions can be screened out from the N image frames, so that the target image frame can better convey the core content of the target video, and thus the image frame of the target video can be more accurately determined.

[0098] See also Figure 3 , Figure 3 The following is a flow chart of a method for generating a video cover provided in an embodiment of the present application. Figure 3 As shown, a method for generating a video cover provided in an embodiment of the present application includes the following steps:

[0099] Step 310: Obtain target video;

[0100] Step 320: extracting multiple image frames from the target video;

[0101] In an embodiment of the present application, after extracting N image frames, open source software (such as OpenCV, etc.) can be used to perform image analysis on each of the N image frames. The analyzed image features may specifically include at least one of the features such as contrast, brightness, clarity, and illumination, and a list is constructed based on the results of the image analysis of the image frame and the image frame for subsequent further screening of the image frame. In addition, in the process of image analysis of the image frame, a face detection algorithm and a text recognition algorithm can also be used to meet the needs of different business usage scenarios. For example, popular science videos have high requirements for text clarity, which can increase the attractiveness to users in different business usage scenarios.

[0102] Step 330: extracting multiple audio segments from the target video;

[0103] Step 340: Determine at least one target image frame from the plurality of image frames based on the plurality of audio segments;

[0104] Step 350: Determine A target image frames that meet the target condition from the at least one target image frame; wherein A is a natural number;

[0105] In the embodiment of the present application, the target condition is a requirement of the video cover, and the target condition includes at least one of a contrast condition, a brightness condition, a clarity condition, and an illumination condition. Specifically, A target image frames that meet the video cover requirement are selected from at least one target image frame.

[0106] Step 360: when A is equal to B, determine the A target image frames as image frames of the video cover; wherein B is a preset number of covers;

[0107] In the embodiment of the present application, if the number A of the determined target image frames is exactly equal to the preset cover number B, the A target image frames can be directly determined as the image frames of the video cover.

[0108] In one embodiment of the present application, A=B=1, and the video cover is a static cover.

[0109] In the embodiment of the present application, if the determined number of target image frames A and the preset number of covers B are both 1, the target image frame can be determined as the video cover of the target video in the form of a static cover. In addition, when the preset number of covers B=1 and the number of target image frames A>1, one target image frame can be selected from the A target image frames as the static cover of the target video.

[0110] Step 370: when A is greater than B, obtain the weight score of each target image frame in the A target image frames; obtain B target image frames with the highest weight scores from the A target image frames as image frames for the video cover;

[0111] In an embodiment of the present application, when the number A of the determined target image frames is greater than the preset number of covers B, further screening can be performed by the weight score of the target image frame. The weight score is related to the target condition. Specifically, the weight score includes at least one of a contrast score, a brightness score, a clarity score, and an illumination score. The specific weight settings for contrast, brightness, clarity, and illumination can refer to the business requirements of the target video.

[0112] In an embodiment of the present application, after determining the weight scores of A target image frames, the top B target image frames with the highest weight scores can be selected from the A target image frames according to the order of the weight scores from high to low, and the B target image frames can be determined as image frames of the video cover.

[0113] In one embodiment of the present application, a difference method may be used to select B target image frames from A target image frames. For example, the first target image frame, the third target image frame, the fifth target image frame, and so on may be selected according to the time point sequence corresponding to the target image frames, and B target image frames may be determined and determined as image frames of the video cover.

[0114] Step 380: When A is less than B, reacquire at least one target image frame; and determine the image frame that meets the target condition among the at least one reacquired target image frame as the image frame of the video cover.

[0115] In an embodiment of the present application, if the number of target image frames does not meet the preset number of covers, the process may return to step 340 to reacquire at least one target image frame, and the number of reacquired target image frames should be greater than the number of target image frames acquired last time. Specifically, the number of image frames and audio segments extracted from the target video may be increased. A target image frames that meet the target condition are then rescreened from at least one target image frame, and it is determined again whether the number of re-determined target image frames is less than the preset number of covers. If the number of re-determined target image frames is still less than the preset number of covers, at least one target image frame is acquired again until the number of determined target image frames meets the requirement of the preset number of covers.

[0116] In one embodiment of the present application, A and B are both integers greater than 1, the video cover is a dynamic cover, and the homepage of the dynamic cover is the target image frame with the highest weight score among the A target image frames.

[0117] In the embodiment of the present application, when the preset cover quantity B is greater than 1, it can be determined that the cover produced by the target video is a dynamic cover, and a dynamic cover homepage can be produced for the dynamic cover, and the homepage is the first image that the user sees when seeing the video cover of the target video. Specifically, the target image frame with the highest weight score can be directly selected from the A target image frames according to the weight scores of the A target image frames, and the target image frame can be determined as the homepage of the dynamic cover.

[0118] In one embodiment of the present application, the target image frame with the earliest time point may be determined as the homepage of the dynamic cover according to the time point sequence corresponding to the A target image frames.

[0119] In the embodiment of the present application, a variety of dynamic effects can be pre-set, and different dynamic effects can be selected according to the business scenario of the target video to generate a richer and more natural dynamic video cover. In addition, the dynamic cover can display the temporal changes and rich content of the video to attract the user's attention. At the same time, the target image frame with the highest weight score in the target image frame can be determined as the homepage of the dynamic cover, which can attract the user's first glance, and through the subsequent dynamic cover, the user can roughly understand the core content of the target video.

[0120] In an embodiment of the present application, the target image frame can be further screened according to the target conditions (at least one of the contrast conditions, brightness conditions, clarity conditions and illumination conditions), which can ensure the quality of the image frame of the video cover and avoid the situation where the user gives up watching the target video due to the quality of the cover.

[0121] See also Figure 4 and Figure 5 , Figure 4 The overall concept diagram of a method for generating a video cover provided in an embodiment of the present application is as follows: Figure 5 A complete flowchart of a method for generating a video cover provided in an embodiment of the present application.

[0122] like Figure 4 As shown, by extracting the image frames and audio segments of the target video, and then extracting features from the image frames and audio segments, and clustering the image frames and audio segments based on the feature extraction results. The image frame clustering results and the audio segment clustering results are further integrated, and the video cover of the target video is determined by judging the target conditions.

[0123] like Figure 5 As shown, a method for generating a video cover provided in an embodiment of the present application includes the following steps:

[0124] Step 510: Acquire target video;

[0125] Step 515: extracting N image frames from the target video;

[0126] Step 520: extracting N audio segments from the target video, wherein the N image frames correspond to the N audio segments one by one, the extracted image frames and audio segments have indexes numbered according to the same rule, and N is a positive integer;

[0127] Step 525: performing clustering processing on the N image frames to obtain K first clustering results;

[0128] In an embodiment of the present application, in step 525, clustering processing is performed on the N image frames to obtain K first clustering results, including: performing image feature extraction on the N image frames to obtain a first feature map; based on the first feature map, calculating the image similarity between any two image frames in the N image frames; based on the image similarity, clustering processing is performed on the N image frames to obtain K first clustering results; the image similarity between any two image frames in the first clustering result is greater than a set first threshold.

[0129] Step 530: performing clustering processing on the N audio segments to obtain S second clustering results;

[0130] In an embodiment of the present application, clustering the N audio segments in step 530 to obtain S second clustering results includes: extracting audio features from the N audio segments to obtain a second feature map; calculating the audio similarity between any two audio segments in the N audio segments based on the second feature map; clustering the N audio segments based on the audio similarity to obtain S second clustering results; the audio similarity between any two audio segments in the second clustering results is greater than a set second threshold.

[0131] Step 535: Determine P first clustering results that meet a first condition from the K first clustering results, where K and P are both positive integers, and P is less than or equal to K;

[0132] In an embodiment of the present application, in step 535, P first clustering results that meet the first condition are determined from the K first clustering results, including: determining the number of image frames in each of the K first clustering results; sorting the K first clustering results from most to least according to the number of image frames to obtain a first sorting result; and obtaining the top P first clustering results from the first sorting result.

[0133] Step 540: Determine Q second clustering results that meet a second condition from the S second clustering results, where S and Q are both positive integers, and Q is less than or equal to S;

[0134] In an embodiment of the present application, in step 540, Q second clustering results that meet the second condition are determined from the S second clustering results, including: determining the number of audio segments in each of the S second clustering results; sorting the K first clustering results from most to least according to the number of audio segments to obtain a second sorting result; and obtaining the top Q second clustering results from the second sorting result.

[0135] Step 545: Based on the Q second clustering results, determine at least one target image frame from the P first clustering results, where the index of the target image frame is in the index of the target audio segment included in the Q second clustering results;

[0136] In an embodiment of the present application, in step 545, based on the Q second clustering results, at least one target image frame is determined from the P first clustering results, including: for each of the P first clustering results, the following operations are performed: respectively determine whether the index of each image frame in the current first clustering result is the same as the index of any target audio segment in the Q second clustering results, and if so, determine that the current image frame is the target image frame.

[0137] In the embodiment of the present application, when the number of the target image frames is less than the preset number of covers, the value of P is increased.

[0138] Step 550: Determine A target image frames that meet the target condition from the at least one target image frame; wherein A is a natural number;

[0139] The target condition includes at least one of a contrast condition, a brightness condition, a clarity condition and an illumination condition.

[0140] Step 555: when A is equal to B, the A target image frames are determined as image frames of the video cover; wherein B is the preset number of covers;

[0141] In the embodiment of the present application, if A=B=1, the video cover is a static cover.

[0142] Step 560: when A is greater than B, obtain the weight score of each target image frame in the A target image frames; obtain B target image frames with the highest weight scores from the A target image frames as image frames for the video cover;

[0143] In an embodiment of the present application, if A and B are both integers greater than 1, the video cover is a dynamic cover, and the homepage of the dynamic cover is the target image frame with the highest weight score among the A target image frames.

[0144] Step 565: When A is less than B, reacquire at least one target image frame; and determine the image frame that meets the target condition among the at least one reacquired target image frame as the image frame of the video cover.

[0145] In an embodiment of the present application, a target video is obtained; multiple image frames are extracted from the target video; multiple audio segments are extracted from the target video; at least one target image frame is determined from the multiple image frames based on the multiple audio segments; and an image frame that meets the target condition in the at least one target image frame is determined as an image frame of the video cover. In this way, in the process of determining the video cover, not only the image information of the video is considered, but also the audio information of the video is considered, and at least one audio segment is used to assist in determining the at least one target image frame. Compared with the related art that only considers the image information to determine the video cover, the determined video cover has higher quality, which solves the problem of poor quality of the video cover determined in the related art.

[0146] It is important to understand that Figures 1 to 5 The explanations of the same or corresponding steps in the above may refer to each other. For example, Figure 1 The explanation of step 110 and step 120 in Figure 3 Step 310 and step 320 in .

[0147] At the same time, it should be understood that a method for generating a video cover provided in an embodiment of the present application may have the following beneficial effects: First, the method provided in an embodiment of the present application is combined with deep learning technology to enrich the user's interactive experience, such as the dynamic cover duration, and the dynamic cover duration may be associated with the number of selected image frames. Second, the method provided in the present application can express video semantics in multiple dimensions by fusing image information and audio information, and integrates the judgment of traditional features to improve the image quality of the cover. Third, the method provided in the present application is flexible in setting conditions, which can make the dynamic cover of the video more diverse and more expressive; fourth, the method provided in the present application can be completed by electronic equipment, and the video cover can be automatically generated after the original video is obtained. No manual intervention design is required in the entire process, which can greatly improve the generation efficiency of the video cover and solve the problem that the related technology cannot be applied to big data business scenarios through manual cover design.

[0148] Figure 6 is a structural block diagram of a video cover generation device provided in an embodiment of the present application. Figure 6 The video cover generation device 600 provided in the embodiment of the present application includes:

[0149] An acquisition module 610 is used to acquire a target video;

[0150] An extraction module 620 is used to extract a plurality of image frames from the target video; and extract a plurality of audio segments from the target video;

[0151] The determination module 630 is used to determine at least one target image frame from the multiple image frames based on the multiple audio segments; and determine an image frame that meets a target condition among the at least one target image frame as an image frame of the video cover.

[0152] In an embodiment of the present application, a target video is obtained; multiple image frames are extracted from the target video; multiple audio segments are extracted from the target video; at least one target image frame is determined from the multiple image frames based on the multiple audio segments; and an image frame that meets the target condition in the at least one target image frame is determined as an image frame of the video cover. In this way, in the process of determining the video cover, not only the image information of the video is considered, but also the audio information of the video is considered, and at least one audio segment is used to assist in determining the at least one target image frame. Compared with the related art that only considers the image information to determine the video cover, the determined video cover has higher quality, which solves the problem of poor quality of the video cover determined in the related art.

[0153] The video cover generation device provided in the embodiment of the present application can achieve Figure 5 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0154] like Figure 7 As shown, the embodiment of the present application also provides an electronic device 700, which can be various types of computers, etc. The electronic device 700 includes: a processor 710 and a memory 720, and the memory 720 stores a program or instruction, and the program or instruction is executed by the processor 710 to implement the steps of any of the methods described above. For example, when the program is executed by the processor 710, the following process is implemented: obtaining a target video; extracting multiple image frames from the target video; extracting multiple audio segments from the target video; based on the multiple audio segments, determining at least one target image frame from the multiple image frames; and determining the image frame that meets the target condition in the at least one target image frame as the image frame of the video cover. In this way, in the process of determining the video cover, not only the image information of the video is considered, but also the audio information of the video is considered. At least one audio segment is used to assist in determining the at least one target image frame. Compared with the related art that only considers the image information to determine the video cover, the determined video cover has higher quality, which solves the problem of poor quality of the video cover determined in the related art.

[0155] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of each embodiment of the method for generating a video cover are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0156] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0157] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0158] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0159] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0161] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for generating a video cover, It is characterized in that include: Get the target video; Extracting multiple image frames from the target video; extracting a plurality of audio segments from the target video; determining at least one target image frame from the plurality of image frames based on the plurality of audio segments; An image frame that meets the target condition among the at least one target image frame is determined as an image frame of the video cover.

2. The method according to claim 1, It is characterized in that The number of the plurality of image frames is N, the number of the plurality of audio segments is N, and the N image frames correspond to the N audio segments one by one; the extracted image frames and audio segments have indexes numbered according to the same rule, and N is a positive integer; The at least one target image frame is determined from clustering results corresponding to the N image frames based on an index of a target audio segment among the N audio segments, and the target audio segment is selected from the clustering results corresponding to the N audio segments.

3. The method according to claim 2, It is characterized in that The step of determining at least one target image frame from the plurality of image frames based on the plurality of audio segments comprises: Performing clustering processing on the N image frames to obtain K first clustering results; Performing clustering processing on the N audio segments to obtain S second clustering results; Determine P first clustering results that meet a first condition from the K first clustering results; Determine Q second clustering results satisfying a second condition from the S second clustering results; Based on the Q second clustering results, determining at least one target image frame from the P first clustering results, where an index of the target image frame is in an index of a target audio segment included in the Q second clustering results; Among them, K, S, P and Q are all positive integers, P is less than or equal to K, and Q is less than or equal to S.

4. The method according to claim 3, It is characterized in that Determining P first clustering results satisfying the first condition from the K first clustering results includes: determining the number of image frames in each of the K first clustering results; sorting the K first clustering results from large to small according to the number of image frames to obtain a first sorting result; and obtaining the top P first clustering results from the first sorting result; The step of determining Q second clustering results satisfying the second condition from the S second clustering results comprises: determining the number of audio segments in each of the S second clustering results; sorting the K first clustering results from large to small according to the number of audio segments to obtain a second sorting result; and obtaining the top Q second clustering results from the second sorting result.

5. The method according to claim 4, It is characterized in that After determining at least one target image frame from the plurality of image frames, the method further includes: When the number of the target image frames is less than the preset number of covers, the value of P is increased.

6. The method according to claim 3, It is characterized in that The clustering process of the N image frames to obtain K first clustering results includes: extracting image features of the N image frames to obtain a first feature map; calculating the image similarity between any two image frames in the N image frames based on the first feature map; clustering the N image frames based on the image similarity to obtain K first clustering results; the image similarity between any two image frames in the first clustering results is greater than a set first threshold; The clustering process of the N audio segments to obtain S second clustering results includes: extracting audio features of the N audio segments to obtain a second feature map; calculating the audio similarity between any two audio segments in the N audio segments based on the second feature map; clustering the N audio segments based on the audio similarity to obtain S second clustering results; the audio similarity between any two audio segments in the second clustering results is greater than a set second threshold.

7. The method according to any one of claims 3 to 6, It is characterized in that The determining at least one target image frame from the P first clustering results based on the Q second clustering results comprises: For each of the P first clustering results, the following operations are performed: determine whether the index of each image frame in the current first clustering result is the same as the index of any target audio segment in the Q second clustering results; if so, determine that the current image frame is the target image frame.

8. The method according to claim 1, It is characterized in that The target condition includes: at least one of a contrast condition, a brightness condition, a clarity condition, and an illumination condition; and determining an image frame that meets the target condition in the at least one target image frame as an image frame of the video cover includes: Determine A target image frames that meet the target condition from the at least one target image frame; wherein A is a natural number; When A is equal to B, the A target image frames are determined as image frames of the video cover; When A is greater than B, a weight score of each target image frame in the A target image frames is obtained; B target image frames with the highest weight scores are obtained from the A target image frames as image frames for the video cover; In the case where A is less than B, reacquire at least one target image frame; determine an image frame that meets the target condition among the at least one target image frame reacquired as an image frame of the video cover; Among them, B is the preset number of covers.

9. The method according to claim 8, It is characterized in that When A=B=1, the video cover is a static cover; when A and B are both greater than 1, the video cover is a dynamic cover, and the homepage of the dynamic cover is the target image frame with the highest weight score among the A target image frames.

10. A device for generating a video cover, It is characterized in that include: An acquisition module, used to acquire a target video; An extraction module, used for extracting multiple image frames from the target video; extracting a plurality of audio segments from the target video; a determination module, configured to determine at least one target image frame from the plurality of image frames based on the plurality of audio segments; An image frame that meets the target condition among the at least one target image frame is determined as an image frame of the video cover.

11. An electronic device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores a program or instruction running on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, It is characterized in that The medium stores a program or an instruction, and when the program or the instruction is executed, the steps of the method according to any one of claims 1 to 9 are implemented.