Video cover generation method, device, electronic device and readable medium

Abstract

The present disclosure discloses a method, device, electronic device and readable medium for generating a video cover. The method includes: extracting at least two key frames in a video, wherein the key frames include feature information displayed in the cover; according to the action correlation of each key frame, the feature information in each key frame is fused into a single image to generate the cover of the video, wherein the action correlation includes correlation or irrelevance. The above technical solution fuses the feature information of multiple key frames into a single image, so that a single static image can display rich video content, occupies less resources and has high efficiency, and when fusing the feature information, the action correlation of each key frame is considered, so the way of generating the video cover is more flexible and diverse.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a video cover generation method, device, electronic device and readable medium. Background Art

[0002] Video cover is a form of displaying the key content of a video. It is also a collection of information that users receive at first glance when browsing the video playback page. It plays an important role in attracting users to watch the video. Usually, a frame of an image in a video can be used as the video cover, but this form is relatively simple and the video cover can reflect very little information, which is not conducive to users quickly understanding the key content of the video.

[0003] In order to show more video content, the video cover can also be a beautiful image designed by humans. This situation is relatively diverse and can also reflect more information, but the design process requires the help of certain professional tools (such as PhotoShop, etc.), and it is impossible to automatically generate a video cover. The whole process is time-consuming and labor-intensive. In some scenarios, multiple frames of images in the video are used to generate a dynamic cover. Generally, the most exciting clip in the video is used. Its expression ability is better than that of a static cover, but the corresponding algorithm complexity is higher. The training of the dynamic cover model generally requires a large amount of labeled data, and the labeling is difficult. The process is also very time-consuming and labor-intensive, and the dynamic cover occupies more storage space than the static cover. In summary, the current video cover generation method is time-consuming and labor-intensive, with high costs and low efficiency in generating video covers. Summary of the invention

[0004] The present invention provides a video cover generation method, device, electronic device and readable medium to display rich video content in the cover and improve the efficiency of generating video cover.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a video cover, comprising:

[0006] Extracting at least two key frames from the video, wherein the key frames include feature information displayed in the cover;

[0007] According to the action correlation of each of the key frames, the feature information in each of the key frames is fused into a single image to generate a cover of the video, wherein the action correlation includes correlation or irrelevance.

[0008] In a second aspect, the embodiment of the present disclosure further provides a video cover generation device, including:

[0009] An extraction module, used to extract at least two key frames from a video, wherein the key frames include feature information of the video;

[0010] A generation module is used to fuse the feature information in each of the key frames into a single image according to the action correlation of each of the key frames to generate a cover of the video, wherein the action correlation includes correlation or irrelevance.

[0011] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including:

[0012] one or more processors;

[0013] A storage device for storing one or more programs;

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the video cover generation method as described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the video cover generation method as described in the first aspect.

[0016] The embodiments of the present disclosure provide a method, device, electronic device and readable medium for generating a video cover. The method includes: extracting at least two key frames in a video, wherein the key frames include feature information displayed in the cover; according to the action correlation of each key frame, the feature information in each key frame is fused into a single image to generate the cover of the video, wherein the action correlation includes correlation or irrelevant. The above technical solution fuses the feature information of multiple key frames into a single image, so that a single static image can display rich video content, occupies less resources and has high efficiency. In addition, when fusing the feature information, the action correlation of each key frame is taken into consideration, and the method of generating the video cover is more flexible and diverse. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 is a flowchart of the video cover generation method in the first embodiment of the present disclosure;

[0019] Figure 2 is a flowchart of a method for generating a video cover in Embodiment 2 of the present disclosure;

[0020] Figure 3 is a schematic diagram of fusing instances of action sequence frames into a single image in the second embodiment of the present disclosure;

[0021] Figure 4 is a schematic diagram of a region of a removed instance in a filling action sequence frame in a second embodiment of the present disclosure;

[0022] Figure 5 is a flowchart of the video cover generation method in the third embodiment of the present disclosure;

[0023] Figure 6 is a schematic diagram of fusing the foreground objects of each key frame into the main frame in the third embodiment of the present disclosure;

[0024] Figure 7 is a flowchart of a method for generating a video cover in a fourth embodiment of the present disclosure;

[0025] Figure 8 is a schematic diagram of splicing image blocks of each key frame in the fourth embodiment of the present disclosure;

[0026] Fig. 9 is a flowchart of a method for generating a video cover in Embodiment 5 of the present disclosure;

[0027] Fig.10 is a schematic diagram of the preset color wheel type in the fifth embodiment of the present disclosure;

[0028] Fig.11 is a schematic diagram of adding description text to a single image in Embodiment 5 of the present disclosure;

[0029] Fig.12 is a schematic diagram of a video cover generation process in Embodiment 5 of the present disclosure;

[0030] Fig.13 is a structural diagram of a video cover generating device in Embodiment 6 of the present disclosure;

[0031] Fig.14 It is a schematic diagram of the hardware structure of the electronic device in the seventh embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0035] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0037] In the following embodiments, each embodiment provides optional features and examples at the same time, and the various features recorded in the embodiments can be combined to form multiple optional solutions, and each numbered embodiment should not be regarded as only one technical solution. In addition, the embodiments in this disclosure and the features in the embodiments can be combined with each other without conflict.

[0038] Embodiment 1

[0039] Figure 1 This is a flowchart of the method for generating a video cover in the first embodiment of the present disclosure. This method can be applied to the case of automatically generating a cover for a video. Specifically, by fusing the feature information of multiple frames in the video into a single image as the cover, rich video content can be displayed in the cover. This method can be executed by a video cover generation device, wherein the device can be implemented by software and / or hardware and integrated in an electronic device. The electronic device in this embodiment can be a computer, a laptop, a server, a tablet computer, a smart phone, or other device with an image processing function.

[0040] like Figure 1 As shown, the video cover generation method in the first embodiment of the present disclosure specifically includes the following steps:

[0041] S110. Extract at least two key frames in the video, wherein the key frames include feature information displayed in the cover.

[0042] In this embodiment, the video includes multiple frames of images, and the video can be shot or uploaded by the user, or downloaded from the network. Key frames mainly refer to frames that can reflect the key content of the video or scene changes in multiple frames of images, such as frames containing the main characters in the video, frames belonging to wonderful clips or classic clips, frames with obvious changes in scenes, and frames containing key actions of characters, etc., which can all be used as key frames. Key frames can be selected by performing image similarity clustering and image quality evaluation on multiple frames in the video, or by identifying actions or behaviors in the video.

[0043] Feature information can be understood as features used to describe the specific content of the video reflected by the key frames, such as the color tone of the key frames, the expressions or action features of the characters in the key frames, real-time subtitles matching the key frames, etc. By displaying the feature information in the key frames on the cover, it can attract viewers and enable them to quickly understand the video content.

[0044] In this embodiment, at least two key frames are extracted from the video. On this basis, different key frames can be used to provide diverse feature information for generating the cover, so that the content displayed in the cover is richer.

[0045] S120. According to the action correlation of each of the key frames, the feature information in each of the key frames is integrated into a single image to generate a cover of the video, wherein the action correlation includes correlation or irrelevance.

[0046] In this embodiment, the action correlation of each key frame can be understood as a property that describes whether the instances in different key frames have completed valid actions or behaviors. If it can be identified from the video that the instances in several frames have completed valid actions or behaviors, then these corresponding frames can be used as key frames, and the action correlation between the key frames is relevant; if no valid actions or behaviors can be identified, then the action correlation is irrelevant. Among them, valid actions or behaviors can refer to actions or behaviors that the machine learning model can automatically identify based on the preset behavior library, such as running, jumping, walking, waving or bending, etc. The preset behavior library stores the feature sequences of related actions or behaviors in multiple frames, so they can be learned and identified by the machine learning model.

[0047] Optionally, the action relevance is not only related to whether a valid action or behavior can be identified, but also to the degree of background difference between the corresponding frames, which may include scene content difference and color tone difference in each frame. For example, in several key frames, although the characters are all running, the scenes of the first few frames are parks, and the scenes of the latter few frames are indoors, which means that the running actions in the video do not occur in the same time period. In this case, the background differences of these key frames are relatively large, and the action relevance is irrelevant; for another example, the scenes of several key frames are both parks, but one is a daytime image and the other is a nighttime image, and the color tones are significantly different, so the action relevance is also irrelevant.

[0048] In this embodiment, the action correlation affects the way of fusing the feature information of each key frame. For example, if the action correlation of each key frame is related, then the character instances can be cut out from each key frame and added to the same background, which can be the background of any key frame or a background generated based on at least two key frames. In this case, a static single image can be used as the cover to show a certain action or behavior performed by the character instance in the video, which effectively reduces the computing resources and storage space occupied compared to the method of using dynamic images to show actions or behaviors; if the action correlation of each key frame is unrelated, multiple key frames can be cropped, scaled, spliced, etc., to fuse all or part of the feature information in each key frame into a single image.

[0049] For another example, for each key frame, the character instances therein can be cut out and added to the same background, which can be the background of any key frame or a background generated based on at least two key frames; if the action correlation of each key frame is related, the character instances can be arranged in sequence in chronological order (for example, from left to right, or from right to left, etc.), and the arrangement position of each character instance in the background is kept consistent with its relative position in the original key frame, so that the shape of the character instance is easier to understand visually; if the action correlation of each key frame is unrelated, the instances in each key frame can be arranged freely, without having to be arranged in sequence in chronological order, and without having to keep the arrangement position consistent with the relative position.

[0050] For another example, for each key frame, you can cut out the character instances therein and add these character instances to the same background. If the action correlation of each key frame is related, the background can be generated based on the background of each key frame, so as to maintain the consistency of style between the background and the background of each key frame during the action, and can restore the background of the action process to the greatest extent, making it easier for viewers to understand the action occurring in the background. If the action correlation of each key frame is unrelated, there is no need to consider the consistency of style between the background and the background during the action, and you can use any key frame or any image outside the video (such as a solid color image, an image uploaded or selected by the viewer, or a template image, etc.) as the background, and arrange the character instances in other key frames here.

[0051] Optionally, the action correlation of each key frame can be determined by using an action sequence recognition algorithm. For example, the human pose recognition (Open-pose) algorithm is used to estimate the pose of a person instance in a video. Specifically, firstly, the position coordinates of the human joint points in each frame image of the video are extracted, and the distance change matrix of the human joint points between two adjacent frames is calculated based on this; then the video is segmented, and the video features are generated using the distance change matrix corresponding to each video segment; finally, the video features are classified using a trained classifier. If it can be identified that the video features corresponding to a video segment belong to the action or feature sequence of the behavior in the preset behavior library, then the frame corresponding to this video segment is a key frame, and the action correlation of each key frame is related. For another example, the instance segmentation algorithm is used to extract the outline of the person in each key frame and express the pose, and the key features of the pose are extracted by a clustering algorithm. Based on these key features, the dynamic time warping (DTW) algorithm is used to complete action recognition.

[0052] The video cover generation method in this embodiment, by fusing the feature information of multiple key frames into a single image, can display rich video content with a single static image, occupying less computing resources and storage space, with high efficiency in generating covers, and can attract viewers so that viewers can quickly understand the video content; in addition, when fusing the feature information of different key frames, the action correlation of each key frame is taken into account, and the action correlation affects the way of fusing the feature information of each key frame to generate a video cover in a more flexible and diverse manner.

[0053] Embodiment 2

[0054] Figure 2 4 is a flowchart of a method for generating a video cover in Embodiment 2 of the present disclosure. Based on the above embodiments, this embodiment specifies the process of generating a video cover when the action correlation is related.

[0055] In this embodiment, extracting at least two key frames from a video includes: based on an action recognition algorithm, identifying action sequence frames in the video, and using the action sequence frames as key frames; wherein the action correlation is related. On this basis, the action sequence frames with correlation can be fused, so as to display the video content about a complete action or behavior in a static cover.

[0056] like Figure 2 As shown, the video cover generation method in the second embodiment of the present disclosure includes the following steps:

[0057] S210: Based on an action recognition algorithm, identify action sequence frames in the video, and use the action sequence frames as key frames.

[0058] In this embodiment, the action recognition algorithm can be used to identify valid action sequence frames from the video, and the character instances in each action sequence frame can express a complete action or behavior in a coherent temporal order. The action recognition algorithm can be implemented by the Temporal Segment Network (TSM) model, which is trained based on the Kinetics-400 data set and can be used to recognize 400 actions, which can meet the needs of identifying and displaying the action of the instance in the cover.

[0059] Optionally, when valid action sequence frames are identified, the degree of background difference between the action sequence frames can be further determined. If the degree of background difference is within the allowable range, the action correlation is determined to be relevant, and each action sequence frame can be further instance segmented and image fused to obtain a cover that can express the action or behavior of the instance.

[0060] S220 , performing instance segmentation on each action sequence frame to obtain feature information of each action sequence frame, where the feature information includes an instance and a background.

[0061] Specifically, the main purpose of instance segmentation is to separate the instances from the background in each action sequence frame, wherein each instance can be integrated into the same background to represent a complete action or behavior; each background can be used to generate a cover background. Optionally, a Seperate Object instances by Location and sizes (SOLO) algorithm is used to perform instance segmentation on each action sequence frame. Specifically, the SOLOv2 algorithm can be used to segment instances according to location and size, which has high accuracy and real-time performance, and can improve the efficiency of generating video covers.

[0062] S230: Generate a cover background according to the background of each action sequence frame.

[0063] In this embodiment, the cover background mainly refers to the background used to arrange the instances in each action sequence frame, and can be generated according to the background of each action sequence frame. For example, the pixel values ​​of the background of each action sequence frame at each position are averaged to obtain the cover background. This method is relatively simple and suitable for the case where there are a large number of action sequence frames; for example, the background corresponding to the first action sequence frame, the last action sequence frame, or the middle action sequence frame with the highest image quality is selected from the background of each action sequence frame as the cover background. This method is also easy to implement, but the integration of the background of each action sequence frame is relatively low; for example, for the background of each action sequence frame, the part of the removed instance is a blank area, and the background of other action sequence frames can be used to fill the blank area of ​​the removed instance in the background, and then one frame is selected from the filled action sequence frames as the cover background, or the average of the filled action sequence frames is taken as the cover background, etc. This method can take into account both the quality and the integration of different backgrounds. On this basis, by comprehensively integrating the characteristics of each action sequence frame, the style consistency of the cover and the background of each key frame during the action is guaranteed, so that the viewer can accurately understand the video content.

[0064] S240: Merge instances of each action sequence frame into the cover background to obtain a single image, and use the single image as the cover of the video.

[0065] In this embodiment, the instances of each action sequence frame are added to the cover background, so that a complete action of multiple frames can be displayed using a single static image. In this process, the instances of each action sequence frame can be added to the corresponding position of the cover background according to the relative position of the instance in the original action sequence frame, so as to ensure that the relative position of each instance is consistent with the position during the action, and have a better visualization effect.

[0066] Figure 3 FIG. 2 is a schematic diagram of fusing an instance of an action sequence frame into a single image in the second embodiment of the present disclosure. Figure 3 The single image shown is the cover of the video, in which five character instances can be derived from five action sequence frames, which express the action of a skateboard jump. In order to make the cover generated based on the action sequence frames clearer and more reasonably laid out, after obtaining a unified background, the instances in each action sequence frame can be arranged in a suitable position of the background. It can be understood that, under normal circumstances, if you want to use five key frames to express the actions of the character instances, you need to make these key frames into dynamic images, which requires a large amount of calculation and occupies a large space. The method of this embodiment uses a static single image to effectively fuse the feature information of multiple action sequence frames and use limited resources to display rich video content.

[0067] Optionally, before generating a cover background based on the background of each action sequence frame, it also includes: selecting an action sequence frame as a reference frame, and determining the affine transformation matrix between each action sequence frame and the reference frame based on a feature point matching algorithm; and aligning the background of each action sequence frame with the background of the reference frame based on the affine transformation matrix.

[0068] Specifically, due to different video shooting angles, jitter or errors, the backgrounds of the action sequence frames are not aligned. Directly using the backgrounds of the action sequence frames to generate the cover background will result in local distortion, deformation or blur, which will affect the accuracy and visual effect of the background. Therefore, before generating the cover background based on the backgrounds of the action sequence frames, an action sequence frame can be selected as a reference frame, and the backgrounds of other action sequence frames can be aligned with the background of the reference frame. The reference frame can be the action sequence frame with the highest image quality, the first action sequence frame, the last action sequence frame, or the action sequence frame in the middle.

[0069] In this embodiment, the affine transformation matrix between each action sequence frame and the reference frame is determined according to the feature point matching algorithm, wherein the affine transformation matrix is ​​used to describe the transformation relationship of the matched feature points from the action sequence frame to the reference frame, and the affine transformation includes linear transformation and translation transformation. The feature point matching algorithm can be a scale-invariant feature transform (SIFT) algorithm. Specifically, the key feature points in the background of each action sequence frame are first extracted. These key feature points will not disappear due to factors such as lighting, scale, and rotation. Then, the key points in the action sequence frame and the reference frame are compared pairwise according to the feature vectors of each key point, and several pairs of feature points that match each other between the action sequence frame and the reference frame are found, thereby establishing a corresponding relationship between the feature points and obtaining the affine transformation matrix.

[0070] Optionally, a cover background is generated based on the instances and backgrounds of each key frame, including: for each action sequence frame, removing the corresponding instance from the action sequence frame, and filling the removed area in the action sequence frame according to feature information of a set action sequence frame in a corresponding area, to obtain a filling result corresponding to the action sequence frame, wherein the set action sequence frame includes action sequence frames in each action sequence frame that are different from the current action sequence frame; and generating a cover background based on the filling results of each action sequence frame.

[0071] In this embodiment, the process of generating the cover background can be divided into two stages. In the first stage, for each area of ​​the removed instance in the action sequence frame, the background of other action sequence frames can be used to fill the area, and the filling result corresponding to the action sequence frame can be obtained. The filling result can be understood as a rough background image; in the second stage, the cover background is generated according to the filling results of each action sequence frame. This stage can be understood as a repair process of the rough background image, and the obtained cover background is relatively fine. For example, the rough background images corresponding to each action sequence frame can be averaged to obtain the cover background.

[0072] Figure 4 is a schematic diagram of the area of ​​the removed instance in the filling action sequence frame in the second embodiment of the present disclosure. Figure 4 As shown, assuming that there are N (N is an integer greater than 2) action sequence frames, the blank character shape area in each action sequence frame represents the area after the character instance is removed, and the position or action of the character instance in different action sequence frames may be different. The feature information of the background after the character instance is removed in action sequence frame 1 is represented by a grid; the feature information of the background after the character instance is removed in action sequence frame 2 is represented by a slash; the feature information of the background after the character instance is removed in action sequence frame 3 is represented by a dot texture; the feature information of the background after the character instance is removed in action sequence frame 4 is represented by a vertical line.

[0073] Taking the filling of the blank area after the person instance is removed from the action sequence frame 1 as an example, in the action sequence frame 2, the person shape shown by the dotted line is the corresponding area, and the feature information represented by the oblique line in this area can be used to fill the blank area after the person instance is removed from the action sequence frame 1, but obviously, the person shape shown by the dotted line in the action sequence frame 2 also contains a part of the blank (because the person instance in the action sequence frame 2 is also removed). Therefore, only using the feature information in the corresponding area in the action sequence frame 2 cannot completely fill the area after the person instance is removed from the action sequence frame 1, and the feature information of the corresponding area in the next action sequence frame can be continued to be used to fill; assuming that the next action sequence frame is the action sequence frame N-1, the feature information represented by the dotted texture in the person shape shown by the dotted line in the action sequence frame N-1 can be used to continue to fill the blank area after the person instance is removed from the action sequence frame 1; but it still cannot be completely filled, so it is also necessary to use the feature information represented by the vertical line in the person shape shown by the dotted line in the action sequence N to continue to fill the blank area after the person instance is removed from the action sequence frame 1, so that the filling result of the action sequence frame 1 can be obtained. In the filling result, the feature information of the oblique line part comes from the corresponding area of ​​the action sequence frame 2, the feature information of the dotted part comes from the corresponding area of ​​the action sequence frame 2, and the feature information of the vertical line part comes from the corresponding area of ​​the action sequence frame 2.

[0074] It can be understood that, in one case, if the feature information of the corresponding area of ​​action sequence frame i (2≤i<N) cannot completely fill the area in action sequence frame 1 after the character instance is removed, then the feature information of the corresponding area of ​​action sequence frame i+1 can continue to be used for filling, until the feature information of the corresponding area of ​​the last action sequence frame is used for filling. Regardless of whether it can be completely filled, the filling operation of action sequence frame 1 can be terminated to obtain the filling result of action sequence frame 1.

[0075] In another case, if the feature information of the corresponding area of ​​action sequence frame i (2≤i<N) can be completely filled, the filling operation on action sequence frame 1 can be ended to obtain the filling result of action sequence frame 1 without using subsequent action sequence frames for filling.

[0076] Based on a similar principle, the filling results of action sequence frames 2 to N can be obtained. Then, in the second stage, the cover background can be generated according to the filling results of the action sequence frames. For example, the filling results of each action sequence frame are averaged, or, this embodiment also provides a method for repairing the filling results (rough background image) of each action sequence frame to further process the instance edge and obtain a cover background with higher accuracy.

[0077] Optionally, in the second stage, the filling results of each action sequence frame are repaired, including:

[0078] Dilation processing is performed on the area of ​​the removed instance in each action sequence frame to expand the area of ​​the removed instance, and the expanded area covers the edge part of the removed instance;

[0079] For the dilated area in the action sequence frame, the features of the corresponding areas in the filling results of other action sequence frames are used to repair it, wherein the repair can refer to using a filling operation similar to the first stage, that is, using the features of the corresponding areas in the filling results of other action sequence frames to fill the dilated area again; the repair can also be using the average of the features of the corresponding areas in the filling results of each action sequence frame to fill the dilated area again, so as to obtain the repair result of the action sequence frame, and finally the repair results corresponding to each action sequence frame are averaged to obtain the cover background, so that the edge of the instance can also be fully utilized to fuse the feature information of other action sequence frames.

[0080] In addition, the repair operation in the second stage can be iterated multiple times until the feature difference between the repair result of any action sequence frame in the current iteration and the repair result in the previous iteration is within the allowable range, then the iteration is stopped. At this time, the repair result has fully integrated the feature information in the background of each action sequence frame, and the edge transition is smooth and the accuracy is higher.

[0081] Exemplarily, the process of iteratively performing the repair operation in the second stage includes:

[0082] In the first iteration, for the expanded area of ​​the removed instance in the filling result Rj (1≤j≤N) of the action sequence frame j obtained in the first stage, the feature information of the corresponding areas in R1, R2...RN is averaged and filled in the expanded area to repair the expanded area of ​​the removed instance in Rj, and the repair result Rj1 of the action sequence frame j is obtained;

[0083] Then, the second iteration is entered. Similarly, for the expanded area of ​​the removed instance in the repair result Rj1 of the action sequence frame j, the feature information of the corresponding areas in R1, R2, ... RN is averaged and filled in the expanded area to repair the expanded area of ​​the removed instance in Rj1, and the repair result Rj2 of the action sequence frame j is obtained;

[0084] And so on, until the specified number of iterations is reached, or until the difference between the repair result of any action sequence frame in any iteration and the repair result of the previous iteration is within the allowable range, then the iteration is stopped, and the repair results of all action sequence frames at this time are averaged to obtain the cover background.

[0085] It should be noted that the filling result obtained in the first stage is actually a rough background image. The secondary filling operation in the second stage can further improve the filling accuracy. The incorrect pixel values ​​in the expansion area will be gradually repaired by the correct pixel values, while the correct pixel values ​​of the background part outside the instance will not change with the iteration, ensuring that the generated cover background fully integrates the feature information of each action sequence frame, and the edge processing effect is better, and the transition between the instance and the background is more natural.

[0086] Optionally, the degree of integration of the instances of each action sequence frame with the cover background decreases in chronological order.

[0087] Specifically, Figure 3 As shown, the five character instances in the cover complete a skateboard jump from take-off, vacancy to landing from right to left. The character instances on the left are closer in time sequence, and the character instance on the far left corresponds to the last action sequence frame. The character instances on the left have a lower degree of integration with the cover background, which can also be understood as lower transparency. On this basis, while displaying multiple action sequence frame instances through a static video cover, the time sequence of each instance can also be reflected, which has the effect of visual persistence, making the displayed action or behavior more specific and vivid.

[0088] The video cover generation method in this embodiment identifies the action sequence frames in the video and adds the instances of each action sequence frame to the cover background, thereby displaying the video content about a complete action or behavior in the static cover, so that the cover generated according to the action sequence frame is clearer and the layout is reasonable; by generating the cover background according to the background of each action sequence frame, the characteristics of each action sequence frame can be integrated to ensure the consistency of the style of the cover and the background of each key frame during the action, so that the viewer can accurately understand the video content; by selecting an action sequence frame as a reference frame and aligning the backgrounds of other action sequence frames with the background of the reference frame, the accuracy and visual effect of the generated background are improved; by obtaining a rough background map of each action sequence frame in the first stage and repairing it in the second stage to further improve the filling accuracy, it is ensured that the generated cover background fully integrates the feature information of each action sequence frame, and the edge processing effect is better, and the transition between the instance and the background is more natural; by setting different fusion degrees with the cover background for the instance of each action sequence frame, while displaying multiple action sequence frame instances through a static video cover, the temporal relationship of each instance can also be reflected, making the displayed action or behavior more specific and vivid.

[0089] Embodiment 3

[0090] Figure 5 4 is a flowchart of a method for generating a video cover in Embodiment 3 of the present disclosure. Based on the above embodiments, this embodiment specifies the process of generating a video cover when the action correlation is irrelevant.

[0091] In this embodiment, extracting at least two key frames from the video includes: clustering the images in the video to obtain at least two categories; extracting corresponding key frames from each category based on an image quality assessment algorithm; wherein the action correlation of each key frame is irrelevant. On this basis, different key frames that are irrelevant to the action or behavior can be used to display video content with significant differences on the cover.

[0092] In this embodiment, according to the action correlation of each key frame, the feature information in each key frame is fused into a single image to generate a video cover, including: when the action correlation is irrelevant, a key frame is selected as the main frame; based on the target recognition algorithm, the feature information in each key frame is identified, and the feature information includes the foreground target; the foreground target in each key frame other than the main frame is fused into the main frame to obtain a single image, and the single image is used as the video cover. On this basis, the foreground targets in different key frames can be fused into the same key frame without considering the difference in the background of different key frames, and the method of generating the cover is more flexible.

[0093] like Figure 5As shown, the video cover generation method in the third embodiment of the present disclosure includes the following steps:

[0094] S310: Cluster the images in the video to obtain at least two categories.

[0095] In this embodiment, clustering can be performed based on the inter-frame similarity of each frame image in the video, such as whether the color tone, scene content or included instances are the same, to provide a basis for extracting key frames, wherein the clustering algorithm is, for example, the K-means algorithm.

[0096] S320: Based on the image quality assessment algorithm, extract corresponding key frames from each category.

[0097] In this embodiment, for each category, the quality of each image can be referenced when selecting the key frame. For example, the Hyper Image Quality Assessment (HyperIQA) algorithm is used to evaluate the quality of the images in each category, and then the key frames corresponding to each category are extracted according to the quality of the images in each category. Since the images in each category are similar, it is sufficient to extract one key frame for each category. The extraction of key frames can be achieved through a pre-trained convolutional neural network (CNN), which can automatically use the image with the best quality as the key frame of the category based on the image quality of the images in a category. By extracting the corresponding key frames according to the category, it is possible to avoid extracting too many key frames for the same category and increasing unnecessary calculation and storage space occupancy, and it can be ensured that the content displayed on the cover is not similar or repeated, thereby maximizing the display of more video content on the cover.

[0098] S330: Select a key frame as a main frame.

[0099] In this embodiment, the main frame can be used to arrange foreground objects in other key frames. The main frame can be a key frame with the best image quality, or it can be the first key frame, the last key frame, or a key frame in the middle.

[0100] S340: Identify feature information in each key frame based on a target recognition algorithm, where the feature information includes a foreground target.

[0101] In this embodiment, the foreground target and its location in each key frame can be identified based on the target recognition algorithm. The target recognition algorithm can be an algorithm using a separate CNN model (You Only Look Once, YOLO), such as the YOLOv5 algorithm, which can predict the category and location of the target using a CNN network and has good real-time performance.

[0102] Optionally, you can first identify the foreground target in each key frame and then select the main frame. Based on the recognition result of the foreground target, you can also select a key frame with a relatively prominent foreground target and a relatively simple and uncluttered background as the main frame to facilitate subsequent fusion with each foreground target.

[0103] S350: Fusing the foreground objects in the key frames except the main frame into the main frame to obtain a single image, and using the single image as the cover of the video.

[0104] In this embodiment, the foreground objects in the key frames other than the main frame are arranged in the main frame to generate a cover. When arranging, the foreground objects can be scaled appropriately, and the positional relationship between each foreground object and the original foreground object of the main frame can be considered in the fusion process to reduce the occlusion of the original foreground object, and each foreground object can be centered or evenly distributed as much as possible.

[0105] Figure 6 FIG. 1 is a schematic diagram of fusing the foreground objects of each key frame into the main frame in the third embodiment of the present disclosure. Figure 6 As shown, the cover includes two foreground objects, where foreground object 1 can be the original foreground object of the main frame, and the scene of the main frame is that foreground object 1 stands on the grass; foreground object 2 can be a foreground object extracted from other key frames, and foreground object 2 is integrated into the scene of the main frame. The two foreground objects are one on the left and one on the right, and are centered on the cover as a whole.

[0106] Optionally, the position of the original foreground object in the main frame can be changed to arrange it reasonably with the foreground objects in other key frames, making the arrangement of all foreground objects more flexible. In addition, the outline of each foreground object can be thickened, colored, etc. to make the foreground object more prominent and easier to attract viewers.

[0107] S360: blurring the background of the single image, where the blurring includes blurring or feathering.

[0108] In this embodiment, in order to further emphasize the foreground target, the background may be blurred to a certain extent, mainly including two types of blurring: blurring and feathering. Blurring can make all background areas have the same blurring degree, while feathering can make the blurring degree of areas closer to the foreground target lower and the blurring degree of areas farther from the foreground target higher.

[0109] The fuzzy processing can be expressed as:

[0110] The feathering process can be expressed as:

[0111] Among them, Iblur Indicates the cover after blurring. feather represents the cover after feathering, Blur(,) represents the Gaussian blur function, I represents the input image, M represents the mask of the foreground target, σ represents the standard deviation of the Gaussian distribution, Represents an element-by-element matrix multiplication operation.

[0112] The video cover generation method in this embodiment uses key frames that are unrelated to actions or behaviors to display video content with greatly different content on the cover, thereby enriching the features displayed in the cover; by extracting corresponding key frames according to categories, it can ensure that the content displayed in the cover is not similar or repeated, thereby maximizing the display of more video content in the cover; by identifying the foreground target in each key frame, and arranging the foreground targets in each key frame except the main frame at appropriate positions in the main frame, a static single image can be used to effectively fuse the feature information of multiple key frames; in addition, by processing the outline of the foreground target and blurring the background of the main frame, the foreground target can be made more prominent, allowing viewers to quickly understand the important content of the video.

[0113] Embodiment 4

[0114] Figure 7 4 is a flowchart of a method for generating a video cover in Embodiment 4 of the present disclosure. Based on the above embodiments, this embodiment specifies the process of generating a video cover when the action correlation is irrelevant.

[0115] In this embodiment, according to the action correlation of each key frame, the feature information in each key frame is fused into a single image to generate a video cover, including: extracting image blocks containing feature information in each key frame when the action correlation is irrelevant; splicing each image block to obtain a single image. On this basis, the feature information in different key frames can be displayed in the cover.

[0116] like Figure 7 As shown, the video cover generation method in the third embodiment of the present disclosure includes the following steps:

[0117] S410: Cluster the images in the video to obtain at least two categories.

[0118] S420: Extract corresponding key frames from each category based on an image quality assessment algorithm.

[0119] S430: Extract image blocks containing feature information from each key frame.

[0120] In this embodiment, the image block in the key frame contains feature information, for example, the image block can reflect the color tone of the key frame, the image block contains the expression or action features of the character in the key frame, the image block contains the foreground target identified based on the target recognition algorithm, or the image block contains real-time subtitles matching the key frame, etc.

[0121] S440: splice the image blocks to obtain a single image, and use the single image as the cover of the video.

[0122] Specifically, the image blocks may be spliced ​​together according to a preset template based on the feature information in each image block and the relative proportion of the content in the image block.

[0123] Figure 8 is a schematic diagram of splicing image blocks of each key frame in the fourth embodiment of the present disclosure. Figure 8 As shown, the cover is composed of four image blocks, and the four image blocks may come from different key frames. It should be noted that this embodiment does not limit the shape of each image block and the template for splicing.

[0124] In the video cover generation method of this embodiment, when the action correlation is irrelevant, the image blocks containing feature information in each key frame are extracted; each image block is spliced ​​to obtain a single image. On this basis, the feature information in different key frames can be displayed in the cover, and the method of generating the cover is more flexible.

[0125] Embodiment 5

[0126] Fig. 9 4 is a flowchart of a method for generating a video cover in Embodiment 5 of the present disclosure. Based on the above embodiments, this embodiment specifies the process of adding a description text to a single image.

[0127] In this embodiment, after the feature information in each key frame is integrated into a single image, the method further includes: determining the hue, saturation and brightness of the description text according to the color value of the single image, wherein the color value is converted from the red green blue (RGB) color mode to the hue saturation value (HSV) color mode; and adding the description text at a specified position in the single image according to the hue, saturation and brightness of the description text.

[0128] In this embodiment, the hue of the description text is determined according to the color value of a single image, including: determining the hue type of the single image and the proportion of each hue type based on a clustering algorithm; using the hue type with the highest proportion as the main hue of the single image; and using the hue with the closest hue value corresponding to the main hue in a specified area of ​​a preset color wheel type as the hue of the description text.

[0129] In this embodiment, the saturation and brightness of the description text are determined according to the color value of a single image, including: determining the saturation of the description text according to the average saturation within a set range around the specified position; determining the brightness of the description text according to the average brightness within a set range around the specified position.

[0130] On this basis, the content of the cover can be further enriched and beautified to enable viewers to understand the video content more quickly. The position, size, color and font of the description text can be determined according to the video style and overall color distribution, so that the overall color matching of the cover is more reasonable and the visual effect is better. Optionally, the font of the description text can also be determined according to the theme of the video and the style of the cover, so that the description text is better integrated with the video content and the cover.

[0131] like Fig. 9 As shown, the video cover generation method in the third embodiment of the present disclosure includes the following steps:

[0132] S510: Extract at least two key frames from the video, where the key frames include feature information displayed on the cover.

[0133] S520: According to the action correlation of each key frame, the feature information in each key frame is integrated into a single image to generate a cover of the video.

[0134] S530, converting the color value of the single image from the red, green, and blue (RGB) color mode to the hue, saturation, and value (HSV) color mode.

[0135] In this embodiment, the color value is converted into the HSV color mode. The HSV color mode is a color model for user perception, focusing on color representation, and can reflect the specific color, color depth, and brightness. The color matching of the description text is determined according to the HSV color mode, so that the description text and the cover are more integrated and the viewer's visual effect is more comfortable.

[0136] For a color, the method of converting from RGB color mode to HSV color mode is as follows: the red, green and blue coordinates of the color are marked as (r, g, b), r, g and b are all real numbers between 0 and 1; let max be equivalent to the maximum of r, g and b, and min be equivalent to the minimum of r, g and b. To find the (h, s, v) value of the color in HSV space, where h∈[0,360) is the hue angle, and s, v∈[0,1] are the saturation and brightness, the following conversion relationship is obtained:

[0137]

[0138] v=max.

[0139] S540: Determine the tone type of the single image and the proportion of each tone type based on a clustering algorithm.

[0140] Specifically, the overall color analysis of a single image is performed based on a clustering algorithm. Specifically, a K-means clustering algorithm may be used. For example, the overall color of a single image is clustered into five classes, and the hue type and proportion of the main color of each class are output.

[0141] S550: The color tone type with the highest proportion is used as the main color tone of a single image.

[0142] S560: Use the hue in the specified area of ​​the preset color wheel type that is closest to the hue value corresponding to the main hue as the hue of the description text.

[0143] In this embodiment, the method for determining the hue of the description text is to calculate the color in the predefined color space that is closest to the main hue of the single image and is located in the specified H color wheel type interval as the hue of the description text.

[0144] Fig.10 Schematic diagram of the preset color wheel type in the fifth embodiment of the present disclosure. Fig.10 As shown, among the eight H color wheel types, you can choose one of them to be the hue that is located in the black area (for example, the difference between the main hue and the main hue is within 10°) and is closest to the hue value corresponding to the main hue as the hue of the description text.

[0145] S570: Determine the saturation of the description text according to the average saturation value within a set range around the designated position.

[0146] In this embodiment, the saturation of the description text is determined based on the average saturation value within a set range around the specified position in a single image, so that the saturation of the description text is as consistent as possible with the surrounding saturation, and the integration is stronger. Specifically, the average saturation value within the set range around the specified position is recorded as The specified position is recorded as the origin of the coordinate system, and the saturation of the description text (recorded as S) can be taken as the saturation corresponding to the golden ratio between the origin and S, that is,

[0147] S580: Determine the brightness of the description text according to the average brightness within a set range around the designated position.

[0148] In this embodiment, the brightness of the description text is determined based on the average brightness within a set range around the specified position in a single image, so that the brightness of the description text and the surrounding saturation are as consistent as possible, and the integration is stronger. Specifically, the average brightness within the set range around the specified position is recorded as The specified position is recorded as the coordinate origin, and the brightness of the text (recorded as V) can be taken as the origin and The brightness corresponding to the golden ratio between

[0149] S590: Add description text at a designated position in the single image according to the hue, saturation, and brightness of the description text.

[0150] Fig.11 FIG. 1 is a schematic diagram of adding description text to a single image in Embodiment 5 of the present disclosure. Fig.11 As shown, the description text is added to the lower right corner of the single image, and may be accompanied by a text box, and its font and color may be determined according to the overall style of the single image. It should be noted that this embodiment does not limit the designated location for adding the description text, and may also be, for example, the lower middle position, the upper left corner, or the upper right corner.

[0151] The method of this embodiment can add description text to a single image that integrates feature information of multiple key frames. This process takes into account the overall color distribution of the cover. The hue of the description text is similar to the main hue of the image, and the saturation and brightness are also adapted to the surrounding images. In addition, the contrast between the description text and the color of the single image can also be considered to strengthen or weaken the description text.

[0152] Fig.12 FIG. 1 is a schematic diagram of the video cover generation process in Embodiment 5 of the present disclosure. Fig.12 As shown, in this embodiment, there are three main ways to generate a video cover:

[0153] Method 1: Identify action sequence frames in the video, and when the action correlation of each key frame is related, perform instance segmentation and image fusion based on the action sequence frames, and fuse the instances in multiple action sequence frames into a generated cover background;

[0154] Method 2: When the action correlation of each key frame is irrelevant, cluster the images in the video and extract the key frames, extract the foreground objects in multiple key frames, and merge them into one of the main frames;

[0155] Method 3: When the action correlation of each key frame is irrelevant, cluster the images in the video and extract the key frames, and splice the image blocks in multiple key frames to obtain a single image.

[0156] Through the above method, the feature information in multiple key frames can be reflected in a single static image, thereby improving the diversity of the cover.

[0157] In addition, for a single image obtained by any of the above methods, the hue, saturation and brightness of the description text can also be determined, and the description text can be added to a specified position in the single image accordingly. The content of the description text can be a representative subtitle or a title generated for a video.

[0158] Optionally, for videos, method one can be used preferentially or by default to generate a cover, that is, when a valid action sequence frame is identified, instance segmentation and image fusion are performed based on the action sequence frame, and the instances in multiple action sequence frames are fused into a uniformly generated cover background; if there are no validly identified action sequence frames, method two or method three is used, that is, key frames are extracted using a clustering algorithm, and then foreground objects or image blocks in the key frames are extracted, and then a cover is generated by foreground object segmentation and fusion or image block splicing.

[0159] It should be noted that the above three methods can also be used in combination. For example, in method one, instances in multiple action sequence frames can also be arranged in one of the action sequence frames (the action sequence frame can be used as the main frame); for example, in method two, foreground targets in multiple key frames can also be arranged in a generated cover background.

[0160] The video cover generation method in this embodiment can further enrich and beautify the content of the cover by adding descriptive text, and the hue, saturation and brightness of the descriptive text can be determined according to the overall color value of a single image, so that viewers can understand the video content more quickly, and the overall color matching of the cover is more reasonable and the visual effect is better; in addition, the color matching of the descriptive text is determined by the HSV color model, which can reflect the specific color, color depth and brightness, so that the description text and the cover are more integrated; the video cover generation method in this embodiment provides a variety of ways to generate covers, which improves the flexibility of generating covers.

[0161] Embodiment 6

[0162] Fig.13 Schematic diagram of the structure of the video cover generation device in the sixth embodiment of the present disclosure. For details not yet fully described in this embodiment, please refer to the above embodiments. Fig.13 As shown, the device comprises:

[0163] An extraction module 610 is used to extract at least two key frames from a video, wherein the key frames include feature information of the video;

[0164] The generating module 620 is used to fuse the feature information in each of the key frames into a single image according to the action correlation of each of the key frames to generate the cover of the video, wherein the action correlation includes correlation or irrelevant.

[0165] The video cover generation device of this embodiment can display rich video content with a single static image by fusing the feature information of multiple key frames into a single image, which occupies few resources and is highly efficient. In addition, when fusing the feature information, the action correlation of each key frame is taken into account, and the method of generating video cover is more flexible and diverse.

[0166] On the basis of the above, the extraction module 610 is specifically used to: identify the action sequence frames in the video based on the action recognition algorithm, and use the action sequence frames as the key frames; wherein the action correlation is correlation.

[0167] Based on the above, the generation module 620 includes:

[0168] a segmentation unit, configured to perform instance segmentation on each of the action sequence frames when the action correlation is related, so as to obtain feature information of each of the action sequence frames, wherein the feature information includes an instance and a background;

[0169] A background generating unit, used for generating a cover background according to the background of each action sequence frame;

[0170] The first fusion unit is used to fuse the instances of each of the action sequence frames into the cover background to obtain a single image, and use the single image as the cover of the video.

[0171] On the basis of the above, the background generation unit includes:

[0172] A filling subunit is used for removing the corresponding instance from each action sequence frame, and filling the removed area in the action sequence frame according to the feature information of the set action sequence frame in the corresponding area, so as to obtain the filling result corresponding to the action sequence frame, wherein the set action sequence frame includes the action sequence frames different from the current action sequence frame in each of the action sequence frames;

[0173] A generating subunit is used to generate the cover background according to the filling results of each action sequence frame.

[0174] On the basis of the above, the device also includes:

[0175] A reference frame selection module, used for selecting an action sequence frame as a reference frame before generating a cover background according to the backgrounds of each action sequence frame, and determining an affine transformation matrix between each action sequence frame and the reference frame according to a feature point matching algorithm;

[0176] An alignment module is used to align the background of each of the action sequence frames with the background of the reference frame according to the affine transformation matrix.

[0177] On the basis of the above, the degree of integration of each instance of the action sequence frame with the cover background decreases in chronological order.

[0178] Based on the above, the extraction module 610 includes:

[0179] A clustering unit, used for clustering the images in the video to obtain at least two categories;

[0180] The extraction unit is used to extract corresponding key frames from each of the categories based on an image quality assessment algorithm; wherein the action correlation of each of the key frames is irrelevant.

[0181] Based on the above, the generation module 620 includes:

[0182] A main frame selection unit, used for selecting a key frame as a main frame when the action correlation is irrelevant;

[0183] An identification unit, configured to identify feature information in each of the key frames based on a target recognition algorithm, wherein the feature information includes a foreground target;

[0184] The second fusion unit is used to fuse the foreground objects in the key frames except the main frame into the main frame to obtain a single image, and use the single image as the cover of the video.

[0185] On the basis of the above, the device also includes:

[0186] The blur module is used to blur the background of a single image after obtaining the single image, wherein the blur processing includes blur processing or feathering processing.

[0187] Based on the above, the generation module 620 includes:

[0188] An image block extraction unit, configured to extract an image block containing the feature information in each of the key frames when the action correlation is irrelevant;

[0189] The splicing unit is used to splice the image blocks to obtain the single image.

[0190] On the basis of the above, the device also includes:

[0191] A text color determination module is used to determine the hue, saturation and brightness of the description text according to the color value of the single image after the feature information in each of the key frames is merged into the single image, wherein the color value is converted from the red, green and blue RGB color mode to the hue, saturation and brightness HSV color mode;

[0192] The text adding module is used to add description text at a designated position in the single image according to the hue, saturation and brightness of the description text.

[0193] Based on the above, the text adding module includes:

[0194] A proportion calculation unit, used to determine the tone type of the single image and the proportion of each tone type based on a clustering algorithm;

[0195] A main color tone determination unit, used to take the color tone type with the highest proportion as the main color tone of the single image;

[0196] The hue determination unit is used to use the hue with the closest hue value to the main hue in the specified area of ​​the preset color wheel type as the hue of the description text.

[0197] Based on the above, the text adding module includes:

[0198] A saturation determination unit, configured to determine the saturation of the description text according to a saturation average value within a set range around the designated position;

[0199] The brightness determination unit is used to determine the brightness of the description text according to the average brightness within a set range around the specified position.

[0200] The above-mentioned video cover generation device can execute the video cover generation method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0201] Embodiment 7

[0202] Fig.14 It is a schematic diagram of the hardware structure of the electronic device in the seventh embodiment of the present disclosure. Fig.14 The schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure is shown. The electronic device 700 in the embodiment of the present disclosure includes but is not limited to a computer, a laptop, a server, a tablet computer or a smart phone and other devices with image processing functions. Fig.14 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0203] like Fig.14As shown, the electronic device 700 may include one or more processing devices (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. One or more processing devices 701 implement the traffic data packet forwarding method provided by the present disclosure. In RAM703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, ROM 702, and RAM703 are connected to each other via a bus 705. An input / output (I / O) interface 704 is also connected to the bus 705.

[0204] Typically, the following devices may be connected to the I / O interface 704: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc., the storage device 708 being used to store one or more programs; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.14 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0205] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0206] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media are, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0207] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0208] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0209] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: extracts at least two key frames in the video, wherein the key frames include feature information displayed in the cover; and according to the action correlation of each of the key frames, fuses the feature information in each of the key frames into a single image to generate a cover for the video, wherein the action correlation includes correlation or irrelevance.

[0210] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0211] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0212] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0213] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0214] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0215] According to one or more embodiments of the present disclosure, Example 1 provides a method for generating a video cover, including:

[0216] Extracting at least two key frames from the video, wherein the key frames include feature information displayed in the cover;

[0217] According to the action correlation of each of the key frames, the feature information in each of the key frames is fused into a single image to generate a cover of the video, wherein the action correlation includes correlation or irrelevance.

[0218] Example 2 According to the method described in Example 1, extracting at least two key frames in the video includes:

[0219] Based on an action recognition algorithm, identifying action sequence frames in the video, and using the action sequence frames as the key frames;

[0220] Among them, the action correlation is correlation.

[0221] Example 3: According to the method of Example 2, the step of fusing the feature information in each of the key frames into a single image based on the action correlation of each of the key frames to generate a cover of the video includes:

[0222] When the action correlation is related, performing instance segmentation on each of the action sequence frames to obtain feature information of each of the action sequence frames, wherein the feature information includes an instance and a background;

[0223] Generate a cover background according to the background of each action sequence frame;

[0224] The instances of the action sequence frames are fused into the cover background to obtain a single image, and the single image is used as the cover of the video.

[0225] Example 4: According to the method described in Example 3, a cover background is generated according to the instance and background of each key frame, including:

[0226] For each action sequence frame, a corresponding instance is removed from the action sequence frame, and the removed area in the action sequence frame is filled according to feature information of a set action sequence frame in a corresponding area, so as to obtain a filling result corresponding to the action sequence frame, wherein the set action sequence frame includes an action sequence frame different from the current action sequence frame in each of the action sequence frames;

[0227] The cover background is generated according to the filling results of each of the action sequence frames.

[0228] Example 5: According to the method of Example 3, before generating a cover background according to the background of each action sequence frame, the method further includes:

[0229] Selecting an action sequence frame as a reference frame, and determining an affine transformation matrix between each of the action sequence frames and the reference frame according to a feature point matching algorithm;

[0230] The background of each of the action sequence frames is aligned with the background of the reference frame according to the affine transformation matrix.

[0231] Example 6: According to the method described in Example 3, the degree of fusion of each instance of the action sequence frame with the cover background decreases in chronological order.

[0232] Example 7 According to the method of Example 1, extracting at least two key frames in the video includes:

[0233] Clustering the images in the video to obtain at least two categories;

[0234] Based on an image quality assessment algorithm, extracting corresponding key frames from each of the categories;

[0235] The action correlation of each key frame is irrelevant.

[0236] Example 8: According to the method of Example 7, the step of fusing the feature information in each of the key frames into a single image based on the action correlation of each of the key frames to generate a cover of the video includes:

[0237] When the action correlation is irrelevant, selecting a key frame as a main frame;

[0238] Identify feature information in each of the key frames based on a target recognition algorithm, wherein the feature information includes a foreground target;

[0239] The foreground objects in the key frames except the main frame are merged into the main frame to obtain a single image, and the single image is used as the cover of the video.

[0240] Example 9 The method according to Example 8, after obtaining the single image, further includes:

[0241] The background of the single image is blurred, and the blurring includes blurring or feathering.

[0242] Example 10: According to the method of Example 7, the step of fusing the feature information in each of the key frames into a single image based on the action correlation of each of the key frames to generate a cover of the video includes:

[0243] In a case where the action correlation is irrelevant, extracting image blocks containing the feature information in each of the key frames;

[0244] The image blocks are spliced ​​together to obtain the single image.

[0245] Example 11 The method according to any one of Examples 1 to 10, after fusing the feature information in each of the key frames into a single image, further includes:

[0246] Determine the hue, saturation and brightness of the description text according to the color value of the single image, wherein the color value is converted from the red, green and blue (RGB) color model to the hue, saturation and brightness (HSV) color model;

[0247] The description text is added at a designated position in the single image according to the hue, saturation and brightness of the description text.

[0248] Example 12: According to the method of Example 11, determining the hue of the description text according to the color value of the single image includes:

[0249] Determine the tone type of the single image and the proportion of each tone type based on a clustering algorithm;

[0250] The color tone type with the highest proportion is taken as the main color tone of the single image;

[0251] The hue in the designated area of ​​the preset color wheel type that is closest to the hue value corresponding to the main hue is used as the hue of the description text.

[0252] Example 13: According to the method described in Example 11, determining the saturation and brightness of the description text according to the color value of the single image includes:

[0253] Determining the saturation of the description text according to the average saturation value within a set range around the designated position;

[0254] The brightness of the description text is determined according to the average brightness within a set range around the designated position.

[0255] Example 14 provides a video cover generation device, including:

[0256] An extraction module, used to extract at least two key frames from a video, wherein the key frames include feature information of the video;

[0257] A generation module is used to fuse the feature information in each of the key frames into a single image according to the action correlation of each of the key frames to generate a cover of the video, wherein the action correlation includes correlation or irrelevance.

[0258] Example 15 provides an electronic device, including:

[0259] one or more processors;

[0260] A storage device for storing one or more programs;

[0261] When the one or more programs are executed by the one or more processors, the one or more processors implement the video cover generation method as described in any one of Examples 1-13.

[0262] Example 16 provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the video cover generation method as described in any one of Examples 1-13.

[0263] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0264] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0265] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the attached example book is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the example book.

Claims

1. A video cover generation method, characterized in that: include: Extracting at least two key frames from the video, wherein the key frames include feature information displayed in the cover; According to the action correlation of each key frame, the feature information in each key frame is merged into a single image to generate a cover of the video, wherein the action correlation includes correlation or irrelevance, and the feature information is a feature used to describe the specific content of the video reflected by the key frame; The key frame is obtained by performing image similarity clustering and image quality assessment on multiple frames of images in the video, or by identifying actions or behaviors in the video; the fusion method of the key frame feature information includes a fusion method in which the correlation is relevant or a fusion method in which the correlation is irrelevant; Wherein, in the case where the action correlation is related, the step of fusing the feature information in each of the key frames into a single image according to the action correlation of each of the key frames to generate the cover of the video includes: Performing instance segmentation on each action sequence frame to obtain feature information of each action sequence frame, wherein the feature information includes an instance and a background; Generate a cover background according to the background of each action sequence frame; Merging instances of each of the action sequence frames into the cover background to obtain a single image, and using the single image as the cover of the video; Wherein, in the case where the action correlation is irrelevant, the step of fusing the feature information in each of the key frames into a single image according to the action correlation of each of the key frames to generate the cover of the video includes: Select a keyframe as the main frame; Identify feature information in each of the key frames based on a target recognition algorithm, wherein the feature information includes a foreground target; The foreground objects in the key frames except the main frame are merged into the main frame to obtain a single image, and the single image is used as the cover of the video.

2. The method according to claim 1, characterized in that The step of extracting at least two key frames from the video includes: Based on an action recognition algorithm, identifying action sequence frames in the video, and using the action sequence frames as the key frames; Among them, the action correlation is correlation.

3. The method according to claim 1, characterized in that Generating a cover background according to the instance and background of each key frame includes: For each action sequence frame, a corresponding instance is removed from the action sequence frame, and the removed area in the action sequence frame is filled according to feature information of a set action sequence frame in a corresponding area, so as to obtain a filling result corresponding to the action sequence frame, wherein the set action sequence frame includes an action sequence frame different from the current action sequence frame in each of the action sequence frames; The cover background is generated according to the filling results of each of the action sequence frames.

4. The method according to claim 1, characterized in that Before generating the cover background according to the background of each action sequence frame, the method further includes: Selecting an action sequence frame as a reference frame, and determining an affine transformation matrix between each of the action sequence frames and the reference frame according to a feature point matching algorithm; The background of each of the action sequence frames is aligned with the background of the reference frame according to the affine transformation matrix.

5. The method according to claim 1, characterized in that The degree of integration of each instance of the action sequence frame with the cover background decreases in chronological order.

6. The method according to claim 1, characterized in that The step of extracting at least two key frames from the video includes: Clustering the images in the video to obtain at least two categories; Based on an image quality assessment algorithm, extracting corresponding key frames from each of the categories; The action correlation of each key frame is irrelevant.

7. The method according to claim 1, characterized in that After getting a single image, it also includes: The background of the single image is blurred, and the blurring includes blurring or feathering.

8. The method according to claim 6, characterized in that The step of fusing the feature information in each of the key frames into a single image according to the action correlation of each of the key frames to generate a cover of the video includes: In a case where the action correlation is irrelevant, extracting image blocks containing the feature information in each of the key frames; The image blocks are spliced ​​together to obtain the single image.

9. The method according to any one of claims 1 to 8, characterized in that: After the feature information in each of the key frames is fused into a single image, the method further includes: Determine the hue, saturation and brightness of the description text according to the color value of the single image, wherein the color value is converted from the red, green and blue (RGB) color model to the hue, saturation and brightness (HSV) color model; The description text is added at a designated position in the single image according to the hue, saturation and brightness of the description text.

10. The method according to claim 9, characterized in that Determining the hue of the description text according to the color value of the single image includes: Determine the tone type of the single image and the proportion of each tone type based on a clustering algorithm; The color tone type with the highest proportion is taken as the main color tone of the single image; The hue in the designated area of ​​the preset color wheel type that is closest to the hue value corresponding to the main hue is used as the hue of the description text.

11. The method according to claim 9, characterized in that Determining the saturation and brightness of the description text according to the color value of the single image includes: Determining the saturation of the description text according to the average saturation value within a set range around the designated position; The brightness of the description text is determined according to the average brightness within a set range around the designated position.

12. A video cover generation device, characterized in that: include: An extraction module, used to extract at least two key frames from a video, wherein the key frames include feature information of the video; A generating module, configured to fuse feature information in each of the key frames into a single image according to the action correlation of each of the key frames, so as to generate a cover of the video, wherein the action correlation includes correlation or irrelevance, and the feature information is a feature used to describe the specific content of the video reflected by the key frame; The key frame is obtained by performing image similarity clustering and image quality assessment on multiple frames of images in the video, or by identifying actions or behaviors in the video; the fusion method of the key frame feature information includes a fusion method in which the correlation is relevant or a fusion method in which the correlation is irrelevant; Wherein, the generating module comprises: a segmentation unit, configured to perform instance segmentation on each action sequence frame when the action correlation is related, to obtain feature information of each action sequence frame, wherein the feature information includes an instance and a background; A background generating unit, used for generating a cover background according to the background of each action sequence frame; A first fusion unit, used for fusing the instances of the action sequence frames into the cover background to obtain a single image, and using the single image as the cover of the video; Wherein, the generating module further includes: A main frame selection unit, used for selecting a key frame as a main frame when the action correlation is irrelevant; An identification unit, configured to identify feature information in each of the key frames based on a target recognition algorithm, wherein the feature information includes a foreground target; The second fusion unit is used to fuse the foreground objects in the key frames except the main frame into the main frame to obtain a single image, and use the single image as the cover of the video.

13. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video cover generation method as described in any one of claims 1-11.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video cover generation method as described in any one of claims 1-11 is implemented.

Images