Video cover generation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video cover generation method. The method comprises the following steps: determining a foreground entity of a video cover based on an input video; selecting a target video frame comprising a foreground entity from the video and determining features of the foreground entity in the target video frame; text information of the video cover is generated, and features of the text information are determined; determining a background image of the video cover; and generating the video cover based on the target video frame, the features of the foreground entity, the character information, the features of the character information and the background image of the video cover. The invention further provides a video cover generation device, electronic equipment, a computer readable storage medium and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device, storage medium, and program product for generating a video cover. Background Art

[0002] The video cover is the first visual element that users see in relation to a video. It is understandable that an eye-catching video cover can quickly attract the user's attention. In other words, the quality of the video cover can directly affect the user's decision to watch the video. Generally speaking, the video cover can reflect the theme and content of the video by using images, colors, and text related to the video to achieve the goal of attracting the user's attention. However, at the current stage, the production of high-quality video covers usually relies on the producer's careful design. For example, the selection of characters for the cover subject, the font and color design of the cover text, the selection of the cover background image and the setting of filters, as well as the arrangement of the above-mentioned cover elements and the presentation of the overall visual effect all require manual operations, which usually require a lot of labor costs and the production efficiency cannot be improved. Therefore, there is an urgent need for an intelligent video cover generation solution. Summary of the Invention

[0003] In view of this, the embodiments of the present disclosure provide a video cover generation method, device, electronic device, storage medium and program product, which can solve or partially solve the above problems to a certain extent.

[0004] In some embodiments of the present disclosure, the video cover generation method described in the embodiments of the present disclosure may include: determining a foreground entity of a video cover based on an input video; determining a target video frame including the foreground entity from the video and determining features of the foreground entity in the target video frame; generating text information of the video cover and determining features of the text information; determining a background image of the video cover; and generating a video cover based on the target video frame, features of the foreground entity, the text information, features of the text information, and the background image.

[0005] In some embodiments of the present disclosure, determining the foreground entity of the video cover based on the input video includes: sampling the video to obtain multiple sampled video frames; performing target detection on the multiple sampled video frames respectively to determine the subject corresponding to each of the multiple sampled video frames; and taking the subject that appears most frequently in the multiple sampled video frames as the foreground entity.

[0006] In some embodiments of the present disclosure, determining a target video frame including the foreground entity from the video and determining the features of the foreground entity in the target video frame includes: in response to determining that the foreground entity is a portrait entity, extracting sampled video frames including the portrait entity from the multiple sampled video frames to generate a candidate video frame set; wherein the candidate video frame set includes at least one candidate video frame including the portrait entity; determining the face frame of each candidate video frame in the candidate video frame set based on the face recognition result; determining the body frame of each candidate video frame in the candidate video frame set based on the body detection result; deleting candidate video frames that do not meet the preset relative position conditions, resolution conditions and / or transparency conditions between the face frame and the body frame from the candidate video frame set; determining the portrait quality of each candidate video frame in the candidate video frame set, selecting the candidate video frame with the highest portrait quality as the target video frame; and using the face frame and the body frame of the target video frame as the features of the foreground entity in the target video frame.

[0007] In some embodiments of the present disclosure, determining a target video frame including the foreground entity from the video and determining the features of the foreground entity in the target video frame further includes: in response to determining that the foreground entity is a non-portrait entity, extracting sampled video frames including the non-portrait entity from the multiple sampled video frames to generate a candidate video frame set; wherein the candidate video frame set includes at least one candidate video frame including the non-portrait entity; determining the target frame of the non-portrait entity in each candidate video frame in the candidate video frame set based on the target detection result; deleting candidate video frames that do not meet pre-set target frame position conditions and / or resolution conditions from the candidate video frame set; determining the non-portrait entity quality of each candidate video frame in the candidate video frame set, selecting the candidate video frame with the highest non-portrait entity quality as the target video frame; and using the target frame of the target video frame as the feature of the foreground entity in the target video frame.

[0008] In some embodiments of the present disclosure, generating text information of a video cover and determining features of the text information include: taking one of the title information corresponding to the video, a first text obtained by performing optical character recognition on the video, and a second text obtained by performing automatic speech recognition on the video, or any combination thereof as input information; generating a first prompt word based on the input information; and inputting the first prompt word into a large language model, and generating text information of the video cover by the large language model; wherein the text information includes at least one paragraph and each paragraph includes at least one word; and taking the number of paragraphs included in the text information and the number of words included in each paragraph as features of the text information.

[0009] In some embodiments of the present disclosure, determining the background image of the video cover includes: using the background of the target video frame as the background image of the video cover.

[0010] In some embodiments of the present disclosure, determining the background image of the video cover includes: taking one of the title information corresponding to the video, a first text obtained by performing optical character recognition on the video, a second text obtained by performing automatic speech recognition on the video, or any combination thereof as input information; generating a second prompt word based on the input information; and inputting the second prompt word into a text-graph model, and generating the background image of the video cover by the text-graph model.

[0011] In some embodiments of the present disclosure, determining the background image of the video cover includes: selecting the background image of the video cover from a pre-set background image library.

[0012] In some embodiments of the present disclosure, generating a video cover based on the target video frame, the features of the foreground entity, the text information, the features of the text information, and the background image includes: matching the target cover from a pre-established cover composition feature library based on the features of the foreground entity and the features of the text information; obtaining the composition features of the target cover; translating and scaling the target video frame so that the features of the foreground entity match the composition features of the target cover; cropping the translated and scaled target video frame according to the specifications of the video cover to obtain a backup video cover; and pasting the text information into the backup video cover to obtain the video cover.

[0013] In some embodiments of the present disclosure, generating a video cover based on the target video frame, the features of the foreground entity, the text information, the features of the text information, and the background image includes: extracting a foreground image corresponding to the foreground entity from the target video frame; matching the target cover from a pre-established cover composition feature library based on the features of the foreground entity and the features of the text information; obtaining the composition features of the target cover; and based on the target cover composition features, pasting the foreground image and the text information into the background image respectively to obtain the video cover.

[0014] In some embodiments of the present disclosure, extracting the foreground image corresponding to the foreground entity from the target video frame includes: extracting the foreground image including the foreground entity from the target video frame by image segmentation; and performing pre-set post-processing on the foreground image.

[0015] In some embodiments of the present disclosure, the method further includes: determining the main color of the background image; selecting a text color matching scheme whose background color is closest to the main color of the background image from a pre-established text color library as the target text color matching scheme adopted by the text information; and rendering the text information using the target text color matching scheme.

[0016] Corresponding to the above-mentioned video cover generation method, an embodiment of the present disclosure further provides a video cover generation device, including:

[0017] A foreground entity confirmation module, configured to determine a foreground entity of a video cover based on an input video;

[0018] a target video frame determining module, configured to determine a target video frame including the foreground entity from the video and determine features of the foreground entity in the target video frame;

[0019] A text information generation module, configured to generate text information for a video cover and determine characteristics of the text information;

[0020] A background image generation module is used to determine the background image of the video cover; and

[0021] A cover generation module is used to generate a video cover based on the target video frame, the characteristics of the foreground entity, the text information, the characteristics of the text information and the background image.

[0022] In addition, an embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned video cover generation method when executing the computer program.

[0023] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned video cover generation method.

[0024] An embodiment of the present disclosure further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the above-mentioned video cover generation method.

[0025] Thus, in the above-mentioned video cover generation method, apparatus, electronic device, storage medium, and program product, the foreground entity contained in the video cover can be first determined based on the input video, and the target video frame can be further determined from the video based on the determined foreground entity. Furthermore, after the text information of the video cover is generated and the background image of the video cover is determined, the video cover can be intelligently generated based on the target video frame, text information, and background image. The above-mentioned method can automatically generate a video cover based on the input video, effectively improving the production efficiency of the video cover while ensuring the quality of the video cover, and effectively solving the problem of low generation efficiency caused by excessive reliance on manual operation in the existing video cover generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A schematic diagram of an exemplary system provided by some embodiments of the present disclosure is shown.

[0028] Figure 2 The implementation process of the video cover generation method described in some embodiments of the present disclosure is shown.

[0029] Figure 3 The specific method of determining the foreground entity of a video cover based on an input video according to an embodiment of the present disclosure is shown.

[0030] Figure 4 The present invention shows an implementation process of a method for determining a target video frame including a foreground entity from a video and determining features of the foreground entity in the target video frame when the foreground entity is a portrait entity, as described in some embodiments of the present disclosure.

[0031] Figure 5 The present invention shows an implementation process of a method for determining a target video frame including a foreground entity from a video and determining features of the foreground entity in the target video frame when the foreground entity is a non-portrait entity, as described in some embodiments of the present disclosure.

[0032] Figure 6 The embodiment of the present disclosure shows the implementation process of the method for generating text information of a video cover and determining the characteristics of the text information.

[0033] Figure 7The implementation process of the method for generating a video cover based on a target video frame, features of a foreground entity, text information, features of the text information, and a background image as described in some embodiments of the present disclosure is shown.

[0034] Figure 8a and Figure 8b An example of matching features for one cover in the cover library is shown separately.

[0035] Figure 9 The specific implementation process of the method for extracting the text color matching scheme of a cover in the cover library described in the embodiment of the present disclosure is shown.

[0036] Figure 10 The embodiment of the present disclosure shows the implementation process of the method for generating a video cover based on a target video frame, features of a foreground entity, text information, features of the text information, and a background image.

[0037] Figure 11 The internal structure of the video cover generation device described in some embodiments of the present disclosure.

[0038] Figure 12 A more specific schematic diagram of the hardware structure of an electronic device described in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0040] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0041] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0042] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0043] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0044] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0045] As used herein, the term "in response to" refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is satisfied. For example, in some cases, a subsequent action may be executed immediately upon the occurrence of the event or the satisfaction of the condition; in other cases, the subsequent action may be executed some time after the occurrence of the event or the satisfaction of the condition.

[0046] As mentioned above, video covers can capture the user's attention by using relevant images, colors, and text to reflect the theme and content of the video. However, at the current stage, the production of high-quality video covers usually relies on the producer's careful design. For example, in some simple scenarios, the producer can select a frame from the video and add some text content to it as the cover. In some complex scenarios, the producer needs to carefully match and design the cover elements extracted from the video to obtain a video cover with higher visual quality. It can be seen that whether the scenario is simple or complex, it relies on understanding the content of the video and manually converting it into cover elements. In addition, the cover elements need to be manually arranged to obtain the best overall visual effect. It can be seen that the production of video covers at this stage requires a lot of labor costs, and production efficiency cannot be improved. Therefore, there is an urgent need for an intelligent video cover generation solution.

[0047] In view of this, the embodiments of the present disclosure provide a video cover generation method, device, electronic device, storage medium and program product, which can realize the automatic extraction of cover elements, the reasonable arrangement of cover elements and even the reasonable matching of text colors, thereby realizing the automatic generation of high-quality video covers.

[0048] For the sake of clarity in description, before describing the specific technical solutions of the embodiments of the present disclosure, the technical terms involved in the embodiments of the present disclosure are first explained.

[0049] A large language model refers to a deep learning model trained using large amounts of text data, enabling it to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on various topics by training on large datasets. The core idea is to learn the patterns and structure of natural language through large-scale unsupervised training, simulating the human language cognition and generation process to a certain extent.

[0050] A text-to-image model is an artificial intelligence technology based on deep learning that can convert natural language descriptions into corresponding images. Its core goal is to generate high-quality, diverse visual content that matches the description by parsing the semantics of the text. One of the current mainstream text-to-image models is the diffusion model. The diffusion model can generate images through a gradual denoising process. Specifically, noise can first be added to the real image to form a series of blurred images, and then the diffusion model can be trained to learn how to gradually restore clear images from these blurred images. Generally speaking, the diffusion model can generate high-quality, detailed images, and excel in diversity and creativity, and can be widely used in fields such as art creation and design.

[0051] Object detection is a core technology in computer vision. It aims to locate and identify multiple objects in an image and output the object category and its bounding box, typically a rectangular box. Typically, object detection is accomplished by using deep learning models (such as YOLO and Faster R-CNN) to scan an image, identify the object's boundaries, and identify the object's category. This object bounding box, typically a rectangular box, is referred to as a target box.

[0052] Human detection is a branch of object detection that aims to automatically identify and locate human objects in static images or video streams. Typically, human detection can be performed by using deep learning models (e.g., YOLO, Faster R-CNN, etc.) to scan images and define the boundaries of the human body, usually as a rectangular box, which can be called a human body box.

[0053] Image segmentation is a core technology in computer vision. Its core goal is to divide an image into non-overlapping regions with similar characteristics (such as grayscale, texture, and color) and assign semantic labels to specific regions. The foreground can refer to the target object or main area of ​​interest in the segmentation task, forming a binary opposition with the background.

[0054] Portrait segmentation is a technical branch of image segmentation, which aims to finely separate human pixels from the background in an image or video, generate a pixel-level mask, and thus achieve pixel-level segmentation.

[0055] Facial recognition is a biometric technology that uses facial features to identify individuals. It uses computer vision to analyze facial images or video streams to automatically verify and identify individuals. Typically, facial recognition locates the face in a static image or video stream, typically as a rectangular frame, also known as a face frame.

[0056] The Otsu algorithm is an adaptive thresholding method for image binarization. It determines the optimal threshold by maximizing the inter-class variance, thereby separating the image into foreground and background. First, the Otsu algorithm calculates the grayscale histogram of the image and counts the number of pixels at each grayscale level based on the histogram. Next, the probability of each grayscale level is calculated based on this data. All possible thresholds are iterated to calculate the inter-class variance of the foreground and background, reflecting the degree of separation between the two. Finally, the threshold that maximizes the inter-class variance is selected as the binarization criterion. The Otsu algorithm is simple and efficient, particularly suitable for images with uniform illumination, and is widely used in the field of image processing.

[0057] The K-means algorithm is a commonly used clustering analysis method that aims to divide data points into K clusters, such that data points within the same cluster are highly similar, while data points between different clusters are less similar. First, the algorithm randomly selects K initial centroids and assigns each data point to the cluster represented by the nearest centroid. Next, the average value of all data points within each cluster is calculated and the centroid is updated to this average value. This process is iterated until the centroid no longer changes significantly or a preset number of iterations is reached. The K-means algorithm is simple to understand and computationally efficient, making it suitable for processing large datasets. It is widely used in fields such as image processing, market segmentation, and social network analysis.

[0058] Frame extraction is a core method in video processing that selectively extracts specific frames (i.e., images) from a continuous video stream for analysis, storage, or compression. Specific methods for frame extraction include uniform frame extraction, key frame extraction, scene transition frame extraction, and adaptive frame extraction.

[0059] Super-resolution processing refers to the process of restoring a higher-resolution image from a given low-resolution image or image sequence using software or hardware algorithms, aiming to enhance image clarity, detail, and visual quality. The core of these super-resolution technologies is the use of digital processing to overcome image blurring caused by acquisition equipment, environmental limitations, or transmission losses, thereby improving spatial resolution.

[0060] Instruction tuning is a specialized fine-tuning method that trains large language models to understand and follow diverse human instructions by inputting natural language instructions and corresponding output examples. Instruction tuning often bridges the gap between pre-trained general capabilities and downstream task applications. Through supervised training from "task instructions to expected outputs," it addresses the problem of pre-trained models possessing extensive knowledge but lacking application skills, transforming them into task executors that follow human instructions.

[0061] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

[0062] like Figure 1 As shown, system 100 may include terminal device 102, terminal device 104, and server 106. A medium (e.g., a network) providing a communication link may be included between terminal device 102, terminal device 104, and server 106. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0063] Exemplarily, the terminal device 102 and the terminal device 104 may be installed with an application (APP) or software that can generate a video cover for the video. These applications or software can usually automatically generate a video cover based on the video input by the user. The terminal device 102 and the terminal device 104 here can be hardware or software. When the terminal device 102 and the terminal device 104 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers (Laptops) and desktop computers (PCs), etc. When the terminal device 102 and the terminal device 104 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. It is not specifically limited here.

[0064] The server 106 can be a server that can generate a video cover for a video, such as a background server that provides support for the application or software on the terminal device 102 or the terminal device 104. The server 106 here can also be hardware or software. When the server 106 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 106 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0065] It should be understood that Figure 1 The numbers of terminal devices, users and servers in the embodiment are merely illustrative. Any number of terminal devices, users and servers may be used as required.

[0066] As an example scenario, server 106 can provide a service for generating video cover for a video. User 112 can submit a video to server 106 using an application on terminal device 102. Server 106 automatically generates a video cover for the video submitted by user 112 via terminal device 102. Server 106 can then feed the generated video cover to user 112 via terminal device 102.

[0067] In another exemplary scenario, the application that can be downloaded and installed from server 106 on terminal device 104 and that can generate video cover can support offline video processing. In this case, user 114 can directly use the application on terminal device 104 to generate a video cover based on the video input by the user. In other words, the specific reasoning process described above can be completed independently and offline by terminal device 104 without the real-time participation of server 106.

[0068] Based on the above system 100, in order to solve the problems in the video cover generation process, the embodiment of the present disclosure provides a video cover generation method. The video cover generation method provided by the embodiment of the present disclosure will be described below with reference to specific embodiments and drawings.

[0069] Figure 2 The implementation process of the video cover generation method described in the embodiment of the present disclosure is shown. Figure 2 As shown, the video cover generation method described in the embodiment of the present disclosure can generally be executed by the above-mentioned terminal devices 102, 104 or can also be executed by the above-mentioned server 106, and can specifically include the following steps.

[0070] In step 210 , a foreground entity of a video cover is determined based on the input video.

[0071] At step 220 , a target video frame including a foreground entity is determined from the video and features of the foreground entity in the target video frame are determined.

[0072] In step 230, video cover text information is generated and features of the text information are determined.

[0073] In step 240, a background image of the video cover is determined.

[0074] In step 250, a video cover is generated based on the target video frame, the features of the foreground entity, the text information, the features of the text information, and the background image.

[0075] Thus, in the above-mentioned video cover generation method, the foreground entity contained in the video cover can be first determined based on the input video, and the target video frame can be further determined from the video based on the determined foreground entity. Furthermore, after the text information of the video cover is generated and the background image of the video cover is determined, the video cover can be intelligently generated based on the target video frame, text information, and background image. The above-mentioned method can automatically generate a video cover based on the input video, effectively improving the production efficiency of the video cover while ensuring the quality of the video cover, and effectively solving the problem of low generation efficiency caused by excessive reliance on manual operation in the existing video cover generation process.

[0076] The following will describe in detail the various steps of the video cover generation method described in the embodiment of the present disclosure with reference to specific examples and drawings.

[0077] For the above step 210, illustratively, Figure 1 Users 112 and 114 input videos through terminal devices 102 and 104. Then, the terminal device 102 or 104, or the terminal device 102 or 104 further uploads the input video to the server 106, and the server 106 determines the foreground entity of the video cover based on the input video.

[0078] Specifically, Figure 3 The specific method of determining the foreground entity of the video cover based on the input video according to the embodiment of the present disclosure is shown. Figure 3 As shown, a specific method for determining a foreground entity of a video cover based on a video may include:

[0079] In step 310, the input video is subjected to frame sampling to obtain a plurality of sampled video frames.

[0080] In the embodiment of the present disclosure, step 310 may specifically perform frame extraction on the input video using one of a variety of frame extraction methods, such as uniform frame extraction, key frame extraction, scene transition frame extraction, and adaptive frame extraction.

[0081] In step 320 , target detection is performed on each of the plurality of sampled video frames to determine a subject corresponding to each of the plurality of sampled video frames.

[0082] In an embodiment of the present disclosure, the above-mentioned target detection can be achieved through a deep learning model. For example, these deep learning models may include but are not limited to YOLO or Faster R-CNN, etc. As mentioned above, the boundaries of the subject in each candidate video frame can be framed and the category of the subject can be identified through target detection. Therefore, in the above-mentioned step 320, the subject corresponding to each sampled video frame can be determined respectively through target detection. In an embodiment of the present disclosure, the above-mentioned subject may include: a portrait entity or a non-portrait entity. Furthermore, for a portrait entity, the embodiment of the present disclosure can further frame a human body frame and a facial frame through human body detection and face recognition, and can also determine the facial features of the portrait entity through face recognition, thereby distinguishing the portrait entity corresponding to the corresponding abstract video frame, that is, different abstract video frames can correspond to different portrait entities. Similarly, non-portrait entities also correspond to different categories, and therefore, different abstract video frames can also correspond to different non-portrait entities.

[0083] In step 330, the subject that appears most frequently in the plurality of sampled video frames is regarded as a foreground entity.

[0084] In the embodiments of the present disclosure, the number of times each subject appears in multiple sampled videos can be counted to obtain the number of occurrences of each subject, and the number of occurrences of the subject can be regarded as the frequency of occurrence of the subject. Specifically, the subject with the most occurrences can be regarded as the subject with the highest frequency of occurrence in multiple sampled video frames. Thus, the subject with the most occurrences in multiple sampled video frames can then be regarded as the foreground entity. It is understood that the foreground entity can also include: a portrait entity or a non-portrait entity.

[0085] From this, we can see that through object detection and data statistics, we can determine the subject that appears most frequently in the video and use it as the foreground entity for the video cover. It can be understood that the foreground entity determined by this method can represent the main subject of the video, such as the main character in the video or an object that reflects the main content of the video. Therefore, this foreground entity can reflect the main theme of the video in one dimension.

[0086] As mentioned above, in the above steps, the foreground entity may include a portrait entity or a non-portrait entity. Based on this, in the above step 220, different methods may be used to determine the target video frame and the features of the foreground entity for different foreground entities.

[0087] Specifically, Figure 4The present invention shows the implementation process of the method for determining a target video frame including a foreground entity and determining the features of the foreground entity in the target video frame when the foreground entity is a portrait entity according to some embodiments of the present invention. Figure 4 As shown, the above method may include:

[0088] In step 410 , sampled video frames including a portrait entity are extracted from a plurality of sampled video frames to generate a set of candidate video frames.

[0089] In an embodiment of the present disclosure, the candidate video frame set may include at least one candidate video frame including the portrait entity. As previously described, in the preceding steps, the subjects corresponding to the multiple sampled video frames can be determined through target detection, specifically through body detection and face recognition, for the portrait entity. Therefore, based on the subjects corresponding to the multiple sampled video frames, the sampled video frames including the portrait entity can be extracted from the multiple sampled video frames as candidate video frames, thereby generating the candidate video frame set.

[0090] In step 420 , a facial frame of each candidate video frame in the candidate video frame set is determined based on the face recognition result.

[0091] In the embodiments of the present disclosure, as previously described, facial recognition can be used to determine the facial frame of each candidate video frame. The facial frame can typically be represented as a rectangular frame, and thus, the facial frame is typically represented using the coordinates of the four vertices of the rectangular frame. It will be appreciated that the coordinates of the four vertices can be used to determine the position and size of the facial frame in the candidate video frame.

[0092] In step 430 , a human body frame of each candidate video frame in the candidate video frame set is determined based on the human body detection result.

[0093] In the embodiments of the present disclosure, as previously described, human body detection can be used to determine a human body frame for each candidate video frame. The human body frame can typically be represented as a rectangular frame, and thus, the human body frame is typically represented using the coordinates of the four vertices of the rectangular frame. It will be appreciated that the coordinates of the four vertices can be used to determine the position and size of the human body frame in the candidate video frame.

[0094] At step 440 , candidate video frames that do not meet preset relative position conditions, resolution conditions, and / or transparency conditions of the face frame and the body frame are deleted from the candidate video frame set.

[0095] In the embodiments of the present disclosure, the relative position condition between the face frame and the body frame is primarily used to define the pose of the portrait in the candidate video frames, thereby screening out candidate video frames in which the portrait's face frame is relatively centered relative to its body frame. Specifically, in some embodiments, the relative position condition between the face frame and the body frame can be set to the positional relationship between the vertical centerline of the face frame and the body frame. For example, the vertical centerline of the face frame should be within the center third of the body frame (between the left and right thirds of the body frame). It will be understood that in actual applications, whether the relative position condition between the face frame and the body frame is met can be specifically determined by the coordinates of the four vertices of the face frame and the body frame. In other words, based on the relative position condition between the face frame and the body frame, video frames in which the face is too far to the left or too far to the right relative to the body (for example, candidate video frames in which the vertical centerline of the face frame is within the left or right third of the body frame) can be removed from the candidate video frame set in step 430. It should be noted that the specific values ​​of the above-mentioned left third to right third are only examples. The embodiments of the present disclosure do not limit the specific values ​​of the above-mentioned thresholds, and they can be flexibly set based on actual needs.

[0096] In an embodiment of the present disclosure, the above-mentioned resolution condition may specifically be a condition that the resolution is higher than a pre-set resolution threshold, for example, a resolution higher than 200. That is, based on the above-mentioned resolution condition, candidate video frames having a resolution less than or equal to the above-mentioned resolution threshold may be deleted from the candidate video frame set in step 430. By setting the above-mentioned resolution condition, candidate video frames with lower resolutions may be deleted from the candidate video frame set, or candidate video frames with higher resolutions may be screened out.

[0097] In the embodiment of the present disclosure, the above-mentioned transparency condition is mainly used to limit the position of the portrait in the candidate video frame, so as to avoid the situation in which the top of the portrait is cropped in the candidate video frame, which affects the visual aesthetics. Based on this, the above-mentioned transparency condition can be set to that the proportion of non-zero pixels in the pixel transparency channel of the upper boundary of the candidate video frame is lower than a pre-set ratio threshold, such as 20%. That is to say, through the above-mentioned resolution condition, the candidate video frames whose proportion of non-zero pixels in the pixel transparency channel of the upper boundary is higher than a pre-set ratio threshold (such as 20%) can be deleted from the candidate video frame set in the above-mentioned step 430, so as to delete the candidate video frames whose tops are cropped from the candidate video frame set.

[0098] Furthermore, for the situation where the left or right side of the portrait in the video frame is cropped, which affects the visual aesthetics, that is, when the proportion of non-zero pixels in the pixel transparency channel of the left boundary of the candidate video frame is higher than a preset ratio threshold or the proportion of non-zero pixels in the pixel transparency channel of the right boundary of the candidate video frame is higher than a preset ratio threshold, the non-transparency channel is gradually transitioned to achieve a fade-out effect.

[0099] In step 450 , the portrait quality of each candidate video frame in the candidate video frame set is determined respectively, and the candidate video frame with the highest portrait quality is selected as the target video frame.

[0100] In the embodiments of the present disclosure, the portrait quality of each candidate video frame in the candidate video frame set can be comprehensively evaluated by integrating indicators of multiple dimensions. For example, in some examples, the portrait quality of the candidate video frame can be evaluated from one of the multiple dimensions such as clarity, illumination uniformity, contrast, facial integrity, and expression detection results (mouth opening detection results or eye opening detection results, etc.) or any combination thereof. Specifically, each candidate video frame can be scored separately on the above-mentioned multiple pre-set dimensions; then, the scores of the multiple dimensions can be fused (for example, by weighted summation or weighted averaging, etc.) to obtain the comprehensive score of each candidate video frame; then, each candidate video frame is sorted based on the comprehensive score; finally, the candidate video frame with the highest comprehensive score is used as the target video frame. It should be noted that the embodiments of the present disclosure do not limit the specific method for determining the portrait quality of each candidate video frame in the candidate video frame set, and any portrait quality evaluation method can be used.

[0101] In step 460 , the face frame of the target video frame and the human body frame of the target video frame are used as features of the foreground entity in the target video frame.

[0102] Figure 5 The present invention shows the implementation process of a method for determining a target video frame including a foreground entity and determining the features of the foreground entity in the target video frame when the foreground entity is a non-portrait entity, as described in some embodiments of the present invention. Figure 5 As shown, the above method may include:

[0103] In step 510 , sampled video frames including non-human entities are extracted from a plurality of sampled video frames to generate a set of candidate video frames.

[0104] Similarly, in an embodiment of the present disclosure, the candidate video frame set includes at least one candidate video frame including a non-human entity. As previously described, in the preceding steps, the subjects corresponding to the multiple sampled video frames can be determined through target detection. Therefore, based on the subjects corresponding to the multiple sampled video frames, sampled video frames including the non-human entity as the foreground system can be extracted from the multiple sampled video frames as candidate video frames, thereby generating the candidate video frame set.

[0105] In step 520 , target frames of non-human entities in the candidate video frames are respectively determined based on target detection.

[0106] In the embodiments of the present disclosure, as previously described, target detection can be used to determine the target frame of the non-human entity in each candidate video. The target frame can typically be represented as a rectangular frame, and thus, the target frame is typically represented using the coordinates of the four vertices of the rectangular frame. It will be appreciated that the coordinates of the four vertices can be used to determine the position and size of the target frame in the candidate video frame.

[0107] In step 530 , candidate video frames that do not meet the preset target frame position condition and / or resolution condition are deleted from the candidate video frame set.

[0108] In the embodiments of the present disclosure, the above-mentioned target frame position condition is mainly used to limit the position of the non-portrait entity in the candidate video frame, so that the candidate video frame in which the non-portrait entity position is relatively centered can be screened out from the candidate video frame. Specifically, in some embodiments, the above-mentioned target frame position condition can be set to the range of the target frame coordinates. It can be understood that whether the target frame position condition is met in actual applications can be determined specifically by the coordinates of the four vertices of the target frame. That is to say, through the above-mentioned target frame position condition, the candidate video frames whose target frames are too far to the left, too far to the right, too far above or too far below can be deleted from the candidate video frame set in the above-mentioned step 530.

[0109] In an embodiment of the present disclosure, the above-mentioned resolution condition may specifically be higher than a preset resolution threshold, for example, a resolution higher than 200. That is, based on the above-mentioned resolution condition, candidate video frames having a resolution less than or equal to the above-mentioned resolution threshold may be deleted from the candidate video frame set in step 530. By setting the above-mentioned resolution condition, candidate video frames with poor quality may be deleted from the candidate video frame set.

[0110] In step 540 , the non-human-portrait entity quality of each candidate video frame in the candidate video frame set is determined respectively, and the candidate video frame with the highest non-human-portrait entity quality is selected as the target video frame.

[0111] In the embodiments of the present disclosure, the non-portrait entity quality of each candidate video frame in the candidate video frame set can also be comprehensively evaluated by integrating indicators of multiple dimensions. For example, in some examples, the non-portrait entity quality of the candidate video frame can be evaluated from one of the multiple dimensions such as clarity, illumination uniformity, contrast, or any combination thereof. Specifically, each candidate video frame can be scored on the above-mentioned multiple pre-set dimensions; then, the scores of the multiple dimensions can be fused (for example, by weighted summation or weighted averaging), and the comprehensive scores of each candidate video frame can be obtained respectively; then, each candidate video frame is sorted based on the comprehensive score; finally, the candidate video frame with the highest comprehensive score is used as the target video frame. It should be noted that the embodiments of the present disclosure do not limit the specific method for determining the non-portrait entity quality of each candidate video frame in the candidate video frame set, and any image quality evaluation method can be used.

[0112] In step 550 , the target frame of the target video frame is used as a feature of the foreground entity in the target video frame.

[0113] It can be seen from this that through the above Figure 4 as well as Figure 5 The method can determine a target video frame from a video that includes a foreground entity for a video cover. Furthermore, the target video frame determined by the method not only has high-quality images, but also includes high-quality foreground entities, making it more suitable for use as a foreground entity in a video cover. Therefore, the video cover generated based on the target video frame is typically also of high quality, more effectively achieving the goal of attracting users.

[0114] Regarding the above step 230, Figure 6 The embodiment of the present disclosure shows the implementation process of the method for generating text information of a video cover and determining the characteristics of the text information. Figure 6 As shown, the above method for generating text information of a video cover may include:

[0115] In step 610 , one of the title information corresponding to the video, the first text obtained by performing optical character recognition on the video, the second text obtained by performing automatic speech recognition on the video, or any combination thereof is used as input information.

[0116] In step 620 , a first prompt word is generated based on the input information.

[0117] In an embodiment of the present disclosure, the first prompt word can be generated based on a pre-set prompt word template. The function of the first prompt word is to instruct the large language model to generate text information for the video cover based on the input information. It can be understood that as the text displayed on the video cover, the text information should generally reflect the main theme of the video, and the expression should be concise, that is, the number of words should not be too many. Therefore, in the first prompt word, the number of paragraphs contained in the text information generated by the large language model and the range of the number of words included in each paragraph can be further limited, so that the large language model can be instructed to generate refined text information.

[0118] In step 630, the first prompt word is input into the large language model, and the large language model generates text information of the video cover.

[0119] In an embodiment of the present disclosure, the above text information may include at least one paragraph and each paragraph may include at least one word.

[0120] Furthermore, in order to enable the large language model to more accurately and effectively generate the text information of the video cover, the large language model can be fine-tuned. The specific method of fine-tuning the large language model is not limited in the embodiments of this disclosure, and can be implemented by referring to the existing large language model fine-tuning method.

[0121] In step 640 , the number of paragraphs included in the text information and the number of words included in each paragraph are used as features of the text information.

[0122] This demonstrates that the above method fully leverages the capabilities of a large language model to generate concise and accurate text information for a video cover that reflects the video's main theme. Therefore, the text displayed on the video cover generated based on the above text information can directly reflect the video's main theme, thereby more effectively attracting users.

[0123] With respect to the above step 240 , embodiments of the present disclosure provide multiple methods for determining the background image of a video cover.

[0124] Specifically, in some embodiments of the present disclosure, the background of the target video frame can be directly used as the background image of the video cover. It can be understood that since in the embodiments of the present disclosure, the foreground entity extracted from the target video frame will ultimately be used as the foreground entity of the video cover, using the background of the target video frame as the background image of the video cover can generally produce a video cover with a more natural foreground and background combination.

[0125] However, in some cases, the background area space of the target video frame may not be sufficient, and it cannot fill the video cover after translation and scaling. In these cases, as an alternative to the above solution, the background image of the video cover can also be determined based on a text image model, or based on a pre-established background image library, or based on background images uploaded by users or searched background images.

[0126] In some examples, one of the title information corresponding to the video, the first text obtained by performing optical character recognition on the video, and the second text obtained by performing automatic speech recognition on the video, or a combination thereof, can first be used as input information; then, a second prompt word is generated based on the input information; finally, the second prompt word is input into the text graph model, and the text graph model generates a background image for the video cover. As mentioned above, the core goal of the text graph model is to generate high-quality, diverse visual content that conforms to the description by parsing the semantics of the text. In an embodiment of the present disclosure, the above-mentioned text graph model can be a diffusion model. In addition, the role of the above-mentioned second prompt word is to instruct the text graph model to generate a background image for the video cover based on the input information. In the above method, the ability of the text graph model can be fully utilized to obtain a background image for the video cover that is more in line with the video content, so that the final generated video cover has higher quality, which helps achieve the goal of attracting users.

[0127] In other embodiments, a background image for the video cover can be selected from a pre-set background image library. Typically, the background images in the background image library can be solid-color images or general-purpose images. Therefore, in this case, an image can be randomly selected from the background image library or based on a pre-set background image selection strategy as the background image for the video cover. This method is simple to implement and consumes very little resources, making it particularly suitable for resource-constrained scenarios.

[0128] In the embodiment of the present disclosure, for background images generated in different ways, the above step 250 can be implemented in various ways.

[0129] Figure 7 The following shows the implementation process of the method for generating a video cover when the background image is obtained from the target video frame according to some embodiments of the present disclosure. Figure 7 As shown, the above method for generating a video cover may specifically include the following steps.

[0130] In step 710, a target cover is matched from a pre-established cover library based on the features of the foreground entity and the features of the text information.

[0131] In the embodiments of the present disclosure, a large number of high-quality video covers can be pre-collected as a cover library. Feature extraction is then performed on each cover in the cover library to obtain matching features and composition features for each cover. The matching features are primarily used for feature matching to determine the target cover, while the composition features primarily reflect the compositional characteristics of the cover and are primarily used to generate video covers.

[0132] Specifically, in an embodiment of the present disclosure, for a cover containing a portrait, the above-mentioned matching features may include: a face frame (usually specifically represented as the coordinates of the four vertices of a rectangular frame), a body frame (usually specifically represented as the coordinates of the four vertices of a rectangular frame), one or more text frames (usually specifically represented as the coordinates of the four vertices of a rectangular frame), and the number of words in each text frame. In some embodiments, the above-mentioned composition features may be consistent with the above-mentioned matching features. In other embodiments, the above-mentioned composition features may further include: a text color matching scheme based on the above-mentioned matching features. Usually, it may specifically include: a matching combination of text color, stroke color, and background color.

[0133] In an embodiment of the present disclosure, for a cover that does not contain a portrait, the above-mentioned matching features may include: a target frame (usually specifically represented as the coordinates of the four vertices of a rectangular frame), one or more text boxes (usually specifically represented as the coordinates of the four vertices of a rectangular frame) and the number of words in each text box. Similarly, in some embodiments, the above-mentioned composition features may be consistent with the above-mentioned matching features. In other embodiments, the above-mentioned composition features may further include: a text color matching scheme (usually specifically a combination of text color, stroke color and background color) on the basis of the above-mentioned matching features. In an embodiment of the present disclosure, for the convenience of description, the face frame, the body frame and the target frame may be collectively referred to as the foreground entity frame. That is, for the case where the foreground entity is a portrait entity, the foreground entity frame includes the face frame and the body frame; for the case where the foreground entity is a non-portrait entity, the foreground entity frame includes the target frame.

[0134] Figure 8a and Figure 8b An example of matching features for a cover in the cover library is shown. Figure 8a and Figure 8b The larger solid rectangular box can represent the human body frame on the cover, the smaller solid rectangular box can represent the face frame on the cover, the dotted box represents the text box, and the number in the text box represents the number of words in the text box.

[0135] It should be noted that the above-mentioned feature extraction of each cover in the cover library can be achieved through target detection, face detection, body detection, optical character recognition and other technologies, and the specific implementation process of these methods will not be described in detail.

[0136] In particular, for text color matching schemes, the embodiments of the present disclosure provide a specific method for extracting text color matching schemes from a cover library. Specifically, Figure 9 The specific implementation process of the method for extracting the text color matching scheme of a certain text box in a certain cover in the cover library according to the embodiment of the present disclosure is shown. Figure 9 As shown, for the above text box on the above cover, perform the following steps.

[0137] In step 910 , the text box is enlarged according to a preset ratio to obtain a target text box.

[0138] In the embodiment of the present disclosure, the purpose of enlarging the text box according to a certain ratio is to further frame a part of the background image around the text box into the text box, so that the background color in the text color matching scheme can be effectively extracted. It should be noted that the above-mentioned pre-set ratio can be arbitrarily set as needed, for example, it can be set to 1.1 or other values, etc. It should be noted that the enlargement described in the above step 910 means enlarging the size of the text box so that the enlarged text box can cover more pixels in the image, rather than enlarging the image of the text in the text box by scaling.

[0139] In step 920 , the target text box is rotated to be horizontal.

[0140] In the embodiment of the present disclosure, the method for rotating the target text box can be implemented by using existing image processing tools. The embodiment of the present disclosure does not limit the specific method of rotation.

[0141] In step 930 , the pixels in the target text box are binarized to obtain a text area mask corresponding to the target text box.

[0142] In an embodiment of the present disclosure, the above-mentioned binarization processing may include: first performing grayscale processing on the pixels in the target text box, then performing binarization processing using the adaptive threshold method in the Otsu algorithm, and using the binary image obtained after the binarization processing as a text area mask.

[0143] Furthermore, in order to ensure the polarity consistency between the foreground and background of the text area (for example, the pixel value of the text is 1 and the pixel value of the background is 0), it will be determined whether it is necessary to perform an inversion operation on the binary image obtained after the binarization operation based on the grayscale distribution of the pixel area at the edge of the target text box. Specifically, when the pixel value of the points in the pixel area at the edge of the target text box exceeds a preset proportion (for example, one-half) of the points is 1, it is considered that the binarization operation of the adaptive threshold sets the pixel value of the text foreground to 0 and the pixel value of the background to 1, and thus an inversion operation is required. That is, the pixel value of each pixel point in the binary image obtained by the binarization operation needs to be replaced from 0 to 1 and from 1 to 0 (the pixel value of the point with a pixel value of 0 is modified to 1, and the pixel value of the point with a pixel value of 1 is modified to 0). This ensures that in the text area mask, the pixel value of the text foreground is 1, and the pixel value of the background is 0.

[0144] In step 940 , a text pixel area is determined based on the text area mask.

[0145] In step 950 , the text image area is dilated to a predetermined extent to determine a stroke area and a background area.

[0146] In the embodiments of the present disclosure, the stroked area can be understood as the border area of ​​the text image area, and can generally be understood as the area obtained by subtracting the text pixel area from the expanded text area obtained by dilating the text image area to a predetermined degree. The background area can be the area of ​​the target text frame excluding the text pixel area and the stroked area, that is, the area of ​​the target text frame excluding the expanded text area.

[0147] In step 960 , the colors of the text pixel area, the stroke area, and the background area are determined respectively as the text color, stroke color, and background color in the text color matching scheme.

[0148] In an embodiment of the present disclosure, a specific method for determining the color of a text pixel region, a stroke region, or a background region may include: for each of the above-mentioned text pixel region, stroke region, or background region, calculating the cluster center of the largest cluster in the pixel value clustering results, and using the pixel value of the largest cluster as the color of the region. Specifically, the above-mentioned clustering may utilize a K-means clustering algorithm, and the specific implementation process is not further described.

[0149] Thus, it can be seen that the above method can obtain high-quality and aesthetically pleasing text color contrasting combinations from a high-quality cover library, and when the above color contrasting combinations are applied to video covers, high-quality and aesthetically pleasing text color contrasting combinations can be obtained for video covers. In addition, in the embodiments of the present disclosure, the text color matching schemes extracted for each cover in the cover library can be further aggregated to form a text color library, which is used to obtain more appropriate text color matching schemes during the generation of video covers.

[0150] After obtaining the matching features of each cover through feature extraction, in the above step 710, the features of the foreground entity and the features of the text information can be matched with the matching features of each cover in the cover library, and the cover in the cover library with the highest similarity to the features of the foreground entity and the features of the text information can be used as the target cover.

[0151] As mentioned above, for a portrait entity, the features of the foreground entity may include: a face frame and a body frame. The features of the text information may specifically include the number of paragraphs contained in the text information (which can be understood as the number of text boxes that need to be generated) and the range of the number of characters contained in each paragraph (which can be understood as the number of characters in each text box). The above information can then be matched with the matching features of each cover in the cover library. Specifically, for a candidate cover in the cover library, the above matching process may include: a body matching process and a text matching process.

[0152] For the human body matching process, the target video frame is first translated so that the center point of the face frame in the target video frame coincides with the center point of the face frame of the candidate cover (face position matching); then, the target video frame is scaled proportionally so that the area of ​​the face frame in the target video frame is the same as the area of ​​the face frame of the candidate cover (face size matching); further, based on the human body frame in the target video frame and the human body frame in the candidate cover, the intersection-and-union ratio of the human body area in the target video frame and the human body area in the candidate cover is calculated (to determine the degree of matching of the human body); if the intersection-and-union ratio is greater than a pre-set intersection-and-union ratio threshold (for example, set to 80%), the human body matching is considered appropriate; otherwise, the human body matching is considered inappropriate.

[0153] In the text matching process, in response to determining that the number of paragraphs in the features of the above text information is the same as the number of text boxes in the candidate cover, and the difference between the number of words included in each paragraph in the features of the above text information and the number of words in each text box in the candidate cover is within a pre-set word count threshold (for example, set to 2), the text match is considered appropriate; otherwise, the text match is considered inappropriate. In particular, for the scenario of a single large word box, the word segmentation technology in natural language processing can be further used to segment the paragraph text. If a text box that conforms to the distribution of single words can be extracted, the single word can be queried and matched separately.

[0154] In the embodiments of the present disclosure, a candidate cover that has both a suitable human body and suitable text can be selected as a target cover. If there are multiple candidate covers that meet the requirements, one can be randomly selected as the target cover; alternatively, the candidate cover with the highest similarity (highest intersection-over-union ratio and / or closest number of words in the text box) can be selected as the target cover.

[0155] For non-portrait entities, the features of the foreground entity may include: a target box. The features of the text information may specifically include the number of paragraphs contained in the text information (which can be understood as the number of text boxes that need to be generated) and the range of the number of characters included in each paragraph (which can be understood as the number of characters in each text box). The above information can then be matched with the matching features of each cover in the cover library. Specifically, for a candidate cover in the cover library, the above matching process may include: a text matching process. The text matching process may be consistent with the text matching process in the above portrait entity example, and will not be repeated here.

[0156] In step 720 , the composition features of the target cover are extracted.

[0157] In an embodiment of the present disclosure, as described above, the target cover composition features may include: a foreground entity frame and a text frame in the target cover. Specifically, for a portrait entity, the foreground entity frame may include a face frame and a body frame, which may be specifically represented as the coordinates of the four vertices of the face frame and the body frame. For a non-portrait entity, the foreground entity frame may include a target frame, which may be specifically represented as the coordinates of the four vertices of the target frame. In other embodiments, the composition features of the target cover may further include a text color matching scheme.

[0158] At step 730, the target video frame is translated and scaled so that the features of the foreground entity match the features of the target cover composition.

[0159] In an embodiment of the present disclosure, the above-mentioned step 730 may specifically include: based on the position of the center point of the foreground entity frame in the target cover (the coordinates of the four vertices of the human body frame or the target frame), translating the target video frame so that the center point of the foreground entity in the target video frame coincides with the center point of the foreground entity in the target cover; then, based on the size of the foreground entity frame in the target cover (the area of ​​the human body frame or the target frame), scaling the target video frame so that the size of the foreground entity frame in the target video frame is adjusted to be substantially consistent with the size of the foreground entity in the target cover. In other words, the matching described in the above-mentioned step 730 mainly refers to matching the position and size of the foreground entity in the target video frame and the target cover.

[0160] In the above situation, two results may occur. If the target video frame after translation and scaling can still cover the video cover (that is, the background area space of the target video frame is sufficient), then the subsequent step 740 can be continued; and if the target video frame after translation and scaling can no longer cover the video cover (that is, the background area space of the target video frame is insufficient), then it is necessary to further use other alternative methods to determine the background image of the video cover. For example, the background image can be generated by a text graph model or a sample image can be selected from a background image library to obtain the background image. The specific method for generating a video cover based on the background image generated by the above alternative method will be described in detail later and will be omitted here for the time being.

[0161] In step 740, the target video frame after translation and scaling is cropped according to the specifications of the video cover to obtain a backup video cover.

[0162] It is understood that the video cover usually has a preset specification, such as a specific length and width, etc. Thus, in the embodiment of the present disclosure, in step 740, the target video frame after translation and scaling can be cropped based on the preset specifications.

[0163] In step 750, the text information is pasted into the backup video cover to obtain the video cover.

[0164] In an embodiment of the present disclosure, the process of pasting text information into a background image may include: determining the text box (including position and size) where the text information is located in the video cover based on the text box in the target cover (the coordinates of the four vertices of the text box); then, rendering the text information based on the determined text color matching scheme; and finally pasting the rendered text information into the video cover that has been pasted into the foreground image to obtain the final video cover. Among them, the above-mentioned determined text color matching scheme can be a text color matching scheme included in the target cover composition features, or it can be a text color matching scheme selected from a text color library. In addition, the font corresponding to the text information can directly use a pre-set font or select a font from the font library as the font corresponding to the above text information.

[0165] Furthermore, to ensure the aesthetic appeal of the video cover, layer priorities can be set. For example, text on the body takes precedence over the portrait, and the portrait itself takes precedence over text within the facial area. Furthermore, to ensure the aesthetic appeal of the video cover, it's often required that the straight edge of the portrait itself be aligned with the bottom of the frame, obscured by text, or covered with a solid-color template.

[0166] It can be seen from this that through the above method, the high-quality target video frames intelligently extracted from the video and the text information intelligently generated based on the video can be synthesized based on the composition characteristics of the target cover, so that the video cover can be automatically generated according to the matching and layout of the cover elements in the target cover, effectively solving the problem that the production, matching and layout of cover elements in the existing video cover generation process mainly rely on manual labor.

[0167] Furthermore, in some embodiments of the present disclosure, in order to avoid the target video frame itself containing text that conflicts with the text information of the video cover, thereby affecting the beauty of the video cover, before executing the above step 750, it can further include: erasing the text in the backup video cover.

[0168] Specifically, the above-mentioned erasing of text in the backup video cover can specifically include: first, performing optical character recognition on the backup video cover to obtain a text box in the backup video cover; using the determined text box as a mask area for the erasing operation; and performing an erasing operation based on the determined mask area to erase the text from the backup video frame.

[0169] Specifically, in the above method, the text area in the backup video frame can be located first by means of connected domain analysis, edge detection or deep learning model, and the coordinates of the text box can be output. Furthermore, a pure white rectangular area (value = 255) can be drawn on a blank canvas according to the detected text box coordinates, and the remaining areas are black (value = 0) to generate a binary mask map. Among them, the white rectangular area can correspond to the above mask area. After the mask area is determined, the content of the mask area (that is, the text box) can be removed from the foreground image, and the surrounding background can be retained. Finally, the pixel filling of the above mask area can be completed by using neighborhood pixel filling technology or deep learning-based repair technology to obtain a backup video cover with the text erased. It can be understood that the backup video cover after the text is erased will only contain the foreground entity and the background, which will not affect the pasting of text information.

[0170] Furthermore, in addition to erasing the text in the backup video cover, the backup video cover can also be super-resolution processed to improve the display effect of the video cover. As mentioned above, the above-mentioned super-resolution processing may refer to the process of restoring a higher-resolution image from a given low-resolution image or image sequence through software or hardware algorithms. In the embodiments of the present disclosure, the above-mentioned super-resolution processing may be achieved through an interpolation-based method, a reconstruction-based method, or a learning-based method. Specifically, the above-mentioned interpolation-based method may refer to a method of directly enlarging an image through a mathematical formula (such as bilinear interpolation). The above-mentioned reconstruction-based method may refer to a method of reconstructing details using the temporal information of a multi-frame image sequence. The above-mentioned learning-based method, especially the method based on deep learning (such as convolutional neural network), may refer to learning the features of a large number of image pairs to achieve more accurate reconstruction and significantly improve the effect.

[0171] Figure 10 The following shows the implementation process of the method for generating a video cover when the background image is not obtained from the target video frame according to some embodiments of the present disclosure. Figure 10 As shown, the above method for generating a video cover may specifically include the following steps.

[0172] Step 1010: extract a foreground image corresponding to the foreground entity from the target video frame.

[0173] In an embodiment of the present disclosure, a foreground image including a foreground entity can be extracted from a target video frame by image segmentation. Then, a series of pre-set post-processing is performed on the foreground image to obtain the foreground image in the video cover.

[0174] Specifically, for non-portrait entities, a pixel-level mask can be generated using a general image segmentation algorithm, thereby achieving pixel-level segmentation of the foreground and background in the target video frame. For portrait entities, a pixel-level mask can be generated using a portrait segmentation algorithm, a branch technology in image segmentation, thereby achieving pixel-level segmentation of the portrait and background in the target video frame. It should be noted that the embodiments of the present disclosure do not limit the specific algorithms of the above-mentioned image segmentation algorithm and portrait segmentation algorithm.

[0175] In an embodiment of the present disclosure, the post-processing may include: erasing text in the foreground image, performing super-resolution processing on the foreground image, etc., or a combination thereof.

[0176] Specifically, the above-mentioned erasing of text in the foreground image may include: first, performing optical character recognition on the foreground image to obtain a text box in the foreground image; using the determined text box as a mask area for the erasing operation; and performing an erasing operation based on the determined mask area to erase the text from the foreground image. Specifically, in the above-mentioned method, the text area in the foreground image can be first located by means of connected domain analysis, edge detection or a deep learning model, and the coordinates of the text box can be output. Furthermore, a pure white rectangular area (value = 255) can be drawn on a blank canvas based on the detected text box coordinates, and the remaining area is black (value = 0) to generate a binary mask image. Among them, the white rectangular area can correspond to the above-mentioned mask area. After the mask area is determined, the content of the mask area (that is, the text box) can be removed from the foreground image, and the surrounding background can be retained. Finally, the pixel filling of the above-mentioned mask area can be completed by using a neighborhood pixel filling technology or a deep learning-based repair technology to obtain the foreground image in the video cover.

[0177] The super-resolution processing mentioned above can refer to the super-resolution processing process in the above embodiment, and will not be fully described here.

[0178] From this, it can be seen that through the above process, a foreground image corresponding to a high-quality foreground entity can be obtained, for example, a high-quality portrait image or a non-portrait entity image. Therefore, using the above foreground image as the foreground in the video cover can obtain a high-quality video cover.

[0179] In step 1020, a target cover is matched from a pre-established cover composition feature library based on the features of the foreground entity and the features of the text information.

[0180] In step 1030, the composition features of the target cover are obtained.

[0181] It should be noted that the specific implementation methods of the above steps 1020 and 1030 can refer to steps 710 and 720, and will not be repeated here.

[0182] In step 1040, based on the composition features of the target cover, the foreground image and text information are pasted into the background image to obtain the video cover.

[0183] In an embodiment of the present disclosure, the process of pasting a foreground image into a background image may include: determining the position and size of the foreground image on the video cover based on the foreground entity frame (the coordinates of the four vertices of the human body frame or the target frame) in the target cover; then, translating and scaling the foreground image based on the determined position and size of the foreground image on the video cover and pasting it into the background image to obtain a video cover with the foreground image pasted into it.

[0184] In the embodiment of the present disclosure, the method of pasting text information into the background image can refer to the above step 750, and will not be repeated here.

[0185] Similar to the previous example, to ensure the aesthetic appeal of the video cover, layer priorities can also be set. For example, the layer priority of text on the body is higher than that of the portrait; and the layer priority of the portrait entity is higher than that of text within the facial area. Furthermore, to ensure the aesthetic appeal of the video cover, it is often required that the straight edge of the portrait entity be aligned with the bottom of the frame, blocked by text, or obscured by a solid color template.

[0186] In addition, as mentioned above, in the above scheme, the text color matching scheme used for the text information on the video cover can be the text color matching scheme carried in the target cover composition feature. However, there may be inconsistencies in the color tone of the background of the video cover generated by the above method and the background image of the target cover, which may result in the text color matching scheme determined based on the target cover being unsuitable for the current background. In response to such situations, some embodiments of the present disclosure provide other methods for determining the text color matching scheme used for text information. In these schemes, a text color matching scheme with a background color that is closest to the main color of the background image of the video cover can be selected from a pre-established text color library as the text color matching scheme used for the text information in the video cover.

[0187] Specifically, in the above embodiment, the main color of the background image can be determined by first clustering the pixel values ​​in the background image, and then calculating the cluster center of the largest cluster in the clustering result, and using the pixel value of the above cluster center as the main color of the background image. Specifically, as mentioned above, the above clustering can adopt the K-means clustering algorithm. In addition, as mentioned above, the above pre-established text color library can include a variety of text color matching schemes, and each text color matching scheme can include: background color, font color and stroke color. In this way, in the above embodiment, the background color of each text color matching scheme in the text color library can be compared with the main color of the above background image to find the most similar target text color matching scheme. And the target text color matching scheme is used as the text color matching scheme used for the text information in the video cover, so as to achieve the best color contrast effect between the text and the background, and further enhance the beauty of the video cover.

[0188] It can be understood that the method for determining the text color matching scheme used for the text information on the above-mentioned video cover can be used as an alternative to using the text color matching scheme in the target cover composition feature, or as a supplement to the above-mentioned scheme. That is, when determining the text color matching scheme used for the text information on the video cover, the above-mentioned method of selecting a text color matching scheme from a pre-established text color library can be directly used, or the text color matching scheme in the target cover composition feature can be used first, and then the degree of similarity between the text color in the text color matching scheme and the main color of the background image of the video cover can be further detected. If the two are not similar, the text color matching scheme can be directly used; and if the two are similar, the above-mentioned method of selecting a text color matching scheme from a pre-established text color library can be used to reselect the text color matching scheme, that is, the composition of the video cover can be fine-tuned to achieve a better color scheme. It should be noted that when judging whether two colors are similar, the judgment can be made according to pre-set conditions. The embodiments of the present disclosure do not limit the specific method for judging whether two colors are similar.

[0189] Corresponding to the above-mentioned video cover generation method, some embodiments of the present disclosure further disclose a video cover generation device. Figure 11 The internal structure of the video cover generation device according to the embodiment of the present disclosure is shown. Figure 11 As shown, the video cover generation device may include the following modules:

[0190] A foreground entity confirmation module 1110 is configured to determine a foreground entity of a video cover based on an input video;

[0191] a target video frame determining module 1120 for determining a target video frame including a foreground entity from a video and determining features of the foreground entity in the target video frame;

[0192] The text information generation module 1130 is used to generate text information for the video cover and determine the characteristics of the text information;

[0193] Background image generation module 1140, for determining a background image for the video cover; and

[0194] The cover generation module 1150 is used to generate a video cover based on the target video frame, the features of the foreground entity, the text information, the features of the text information and the background image.

[0195] It should be noted that the implementation methods of each module in the above-mentioned video cover generation device and the specific technical effects that can be achieved can refer to the implementation methods of each step in the above-mentioned embodiment, and will not be repeated here.

[0196] As can be seen from this, the above-mentioned video cover generation device can first determine the foreground entity contained in the video cover based on the input video, and further determine the target video frame from the video based on the determined foreground entity. Furthermore, after generating the text information of the video cover and determining the background image of the video cover, the video cover can be intelligently generated based on the target video frame, text information, and background image. The above-mentioned method can automatically generate video covers based on the input video, effectively improving the production efficiency of video covers while ensuring the quality of the video covers, and effectively solving the problem of low generation efficiency caused by excessive reliance on manual operation in the existing video cover generation process.

[0197] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the video cover generation method described in any of the above embodiments is implemented.

[0198] Figure 12 1 is a schematic diagram showing the hardware structure of a more specific electronic device provided in this embodiment. The device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are communicatively connected to each other within the device via the bus 2050.

[0199] The processor 2010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0200] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 2010.

[0201] The input / output interface 2030 is used to connect input / output devices to enable information input and output. Input / output devices can be configured as components within the device or externally connected to the device to provide corresponding functions. Input devices may include microphones and various sensors, while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0202] The communication interface 2040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0203] The bus 2050 comprises a path for transmitting information between the various components of the device (eg, the processor 2010 , the memory 2020 , the input / output interface 2030 , and the communication interface 2040 ).

[0204] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0205] The electronic device of the above embodiment is used to implement the corresponding video cover generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0206] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the video cover generation method described in any of the above embodiments.

[0207] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0208] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the video cover generation method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0209] Based on the same inventive concept, corresponding to the video cover generation method of any of the above embodiments, the present disclosure also provides a computer program product comprising computer program instructions. In some embodiments, when the computer program instructions are executed on a computer, the computer executes each step of each embodiment of the video cover generation method. For each step in each embodiment of the video cover generation method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0210] The computer program product of the above embodiment is used to enable the processor to execute the video cover generation method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0211] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0212] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0213] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0214] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for generating a video cover, comprising: Determining a foreground entity of a video cover based on an input video; determining a target video frame including the foreground entity from the video and determining features of the foreground entity in the target video frame; generating text information for a video cover and determining characteristics of the text information; Determine the background image of the video cover; as well as A video cover is generated based on the target video frame, the features of the foreground entity, the text information, the features of the text information, and the background image.

2. The method according to claim 1, wherein The determining of the foreground entity of the video cover based on the input video includes: Sampling frames of the video to obtain multiple sampled video frames; Performing target detection on each of the plurality of sampled video frames to determine a subject corresponding to each of the plurality of sampled video frames; and The subject that appears most frequently in the multiple sampled video frames is used as the foreground entity.

3. The method according to claim 1, wherein Determining a target video frame including the foreground entity from the video and determining a feature of the foreground entity in the target video frame includes: In response to determining that the foreground entity is a portrait entity, Extracting a sampled video frame including the human-like entity from the plurality of sampled video frames to generate a candidate video frame set; wherein the candidate video frame set includes at least one candidate video frame including the human-like entity; Determine the facial frame of each candidate video frame in the candidate video frame set based on the face recognition result; Determine the human body frame of each candidate video frame in the candidate video frame set based on the human body detection result; Deleting candidate video frames that do not meet preset relative position conditions, resolution conditions, and / or transparency conditions of the face frame and the body frame from the candidate video frame set; Determining the portrait quality of each candidate video frame in the candidate video frame set respectively, and selecting the candidate video frame with the highest portrait quality as the target video frame; and The face frame of the target video frame and the human body frame of the target video frame are used as features of the foreground entity in the target video frame.

4. The method according to claim 3, wherein: Determining a target video frame including the foreground entity from the video and determining a feature of the foreground entity in the target video frame further comprises: In response to determining that the foreground entity is a non-human-portrait entity, Extracting a sampled video frame including the non-human-like entity from the plurality of sampled video frames to generate a candidate video frame set; wherein the candidate video frame set includes at least one candidate video frame including the non-human-like entity; Determine the target frame of the non-human-like entity in each candidate video frame in the candidate video frame set based on the target detection result; Deleting candidate video frames that do not meet preset target frame position conditions and / or resolution conditions from the candidate video frame set; respectively determining the non-human-portrait entity quality of each candidate video frame in the candidate video frame set, and selecting the candidate video frame with the highest non-human-portrait entity quality as the target video frame; and The target frame of the target video frame is used as a feature of the foreground entity in the target video frame.

5. The method according to claim 1, wherein Generating text information of the video cover and determining the characteristics of the text information includes: Using as input information one of title information corresponding to the video, a first text obtained by performing optical character recognition on the video, a second text obtained by performing automatic speech recognition on the video, or any combination thereof; generating a first prompt word based on the input information; Inputting the first prompt word into a large language model, and generating text information of the video cover by the large language model; wherein the text information includes at least one paragraph and each paragraph includes at least one word; and The number of paragraphs included in the text information and the number of characters included in each paragraph are used as features of the text information.

6. The method according to claim 1, wherein The determining of the background image of the video cover includes: using the background of the target video frame as the background image of the video cover.

7. The method according to claim 1, wherein The background image for determining the video cover includes: Using as input information one of title information corresponding to the video, a first text obtained by performing optical character recognition on the video, a second text obtained by performing automatic speech recognition on the video, or any combination thereof; generating a second prompt word based on the input information; and The second prompt word is input into a text-image model, and the text-image model generates a background image of the video cover.

8. The method according to claim 1, wherein Determining the background image of the video cover includes: selecting the background image of the video cover from a preset background image library.

9. The method according to claim 6, wherein: Generating a video cover based on the target video frame, the feature of the foreground entity, the text information, the feature of the text information, and the background image includes: Matching a target cover from a pre-established cover composition feature library based on the features of the foreground entity and the features of the text information; Acquiring composition features of the target cover; translating and scaling the target video frame so that features of the foreground entity match features of the target cover composition; cropping the target video frame after translation and scaling according to the specifications of the video cover to obtain a backup video cover; and Paste the text information into the backup video cover to obtain the video cover.

10. The method according to claim 9, further comprising: Determine the main color of the background image; Selecting a text color matching scheme whose background color is closest to the main color of the background image from a pre-established text color library as a target text color matching scheme used by the text information; as well as The text information is rendered using the target text color matching scheme.

11. The method according to claim 7 or 8, wherein: Generating a video cover based on the target video frame, the feature of the foreground entity, the text information, the feature of the text information, and the background image includes: Extracting a foreground image corresponding to the foreground entity from the target video frame; Matching a target cover from a pre-established cover composition feature library based on the features of the foreground entity and the features of the text information; Obtaining composition features of the target cover; and Based on the target cover composition features, the foreground image and the text information are pasted into the background image respectively to obtain the video cover.

12. The method according to claim 11, wherein Extracting the foreground image corresponding to the foreground entity from the target video frame includes: extracting a foreground image including the foreground entity from the target video frame by image segmentation; and Performing a preset post-processing on the foreground image.

13. The method according to claim 11, further comprising: Determine the main color of the background image; Selecting a text color matching scheme whose background color is closest to the main color of the background image from a pre-established text color library as a target text color matching scheme used by the text information; as well as The text information is rendered using the target text color matching scheme.

14. A video cover generation device, comprising: A foreground entity confirmation module, configured to determine a foreground entity of a video cover based on an input video; a target video frame determining module, configured to determine a target video frame including the foreground entity from the video and determine features of the foreground entity in the target video frame; A text information generation module, configured to generate text information for a video cover and determine characteristics of the text information; Background image generation module, used to determine the background image of the video cover; as well as A cover generation module is used to generate a video cover based on the target video frame, the characteristics of the foreground entity, the text information, the characteristics of the text information and the background image.

15. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video cover generation method according to any one of claims 1 to 13 when executing the computer program.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the video cover generation method according to any one of claims 1 to 13.

17. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the video cover generation method according to any one of claims 1 to 13.

Images