Video cover generation method, electronic equipment, storage medium and program product

Abstract

The invention provides a video cover generation method, electronic equipment, a storage medium and a program product, and the method comprises the steps: carrying out the first image feature extraction of a plurality of video frames of a target video, and obtaining a plurality of corresponding first image features; furthermore, performing text feature extraction on the video text information of the target video to obtain corresponding text features; selecting a plurality of candidate video cover frames from the plurality of video frames based on similarities between the plurality of first image features and the text features; detecting at least one main body object respectively corresponding to the plurality of candidate video cover frames, and cutting the plurality of candidate video cover frames based on the at least one main body object to generate a plurality of cut candidate video cover frames; and determining a target video cover frame of the target video from the plurality of cut candidate video cover frames. Therefore, the cover frame of the target video can be closely related to the content of the target video, the attention of the user is attracted, and the click rate of the user to the target video is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a video cover generation method, electronic device, storage medium, and program product. Background Art

[0002] With the development of Internet technology, more and more videos are being spread on the Internet. In order to help users understand the content of the video, a video cover frame can be generated for the video and displayed to the user. The user selects the video to watch based on the video cover frame. Therefore, how to generate a video cover frame that is closely related to the content of the video and attracts the user's attention, thereby increasing the user's click rate on the video, is an urgent problem to be solved. Summary of the invention

[0003] In view of this, an embodiment of the present application provides a video cover generation solution to at least partially solve the above-mentioned problem.

[0004] According to a first aspect of an embodiment of the present application, a video cover generation method is provided, comprising: performing first image feature extraction on multiple video frames of a target video respectively to obtain multiple corresponding first image features; and performing text feature extraction on video text information of the target video to obtain corresponding text features; selecting multiple candidate video cover frames from the multiple video frames based on the similarities between the multiple first image features and the text features respectively; detecting at least one main object corresponding to each of the multiple candidate video cover frames, and cropping the multiple candidate video cover frames based on at least one main object to generate multiple cropped candidate video cover frames, the multiple cropped candidate video cover frames respectively containing at least one corresponding main object; and determining a target video cover frame of the target video from the multiple cropped candidate video cover frames.

[0005] According to the second aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect.

[0006] According to a third aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0007] According to a fourth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the method described in the first aspect.

[0008] According to the video cover generation scheme provided in the embodiment of the present application, on the one hand, the selection of multiple candidate video cover frames utilizes the similarity between multiple first image features and text features, which can make the multiple candidate video cover frames closely related to the content of the target video; on the other hand, the cropped multiple candidate video cover frames respectively contain at least one corresponding main object, which can make the cropped multiple candidate video cover frames highlight the main object, thereby attracting user attention and improving user click-through rate of the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0010] Figure 1 An exemplary system for the method for generating a video cover applicable to the embodiment of the present application;

[0011] Figure 2 It is a flowchart of a method for generating a video cover provided in an embodiment of the present application;

[0012] Figure 3 It is a schematic diagram of a process of cropping candidate video covers provided by an embodiment of the present application;

[0013] Figure 4 It is a schematic diagram of a process of determining a target video cover frame provided by an embodiment of the present application;

[0014] Figure 5 It is a detailed flowchart of a video cover generation method provided in an embodiment of the present application;

[0015] Figure 6 This is a schematic diagram of a scenario for generating a short video cover provided by an embodiment of the present application;

[0016] Figure 7 It is a structural schematic diagram of a video cover generation device provided in an embodiment of the present application;

[0017] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0021] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0022] In addition, the term "and / or" in the embodiments of the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0023] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0024] Figure 1 This is an exemplary system applicable to the video cover generation method of the embodiment of the present application. Figure 1 As shown, the system 100 may include a cloud service end 102, a communication network 104 and / or one or more user devices 106. Figure 1 The example in FIG. 1 is a plurality of user devices.

[0025] The cloud server 102 may be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to a distributed storage system device, a server cluster, a computing cloud server cluster, etc. In some embodiments, the cloud server 102 may perform any suitable function. For example, in some embodiments, the cloud server 102 may be used to generate a video cover frame of a target video. Further, the cloud server 102 may send the generated video cover frame to the user device 106.

[0026] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0027] User device 106 may include any one or more user devices suitable for interacting with a user and capable of presenting a video cover frame. In some embodiments, user device 106 may include any suitable type of device. For example, user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of user device.

[0028] It can be understood that the video cover frame in the embodiment of the present application may refer to: an image that is representative of the video and is generated based on a target video.

[0029] With the development of Internet technology, more and more videos are being spread on the Internet. In order to help users understand the content of the video, a video cover frame that is representative of the video can be generated for the video and displayed to the user. The user selects the video to watch based on the video cover frame. Therefore, how to generate a video cover frame that is closely related to the content of the video and attracts the user's attention, thereby increasing the user's click-through rate on the video, is an urgent problem to be solved.

[0030] Based on this, an embodiment of the present application provides a video cover generation method. On the one hand, the selection of multiple candidate video cover frames utilizes the similarity between multiple first image features and text features, which can make the multiple candidate video cover frames closely related to the content of the target video; on the other hand, the cropped multiple candidate video cover frames respectively contain at least one corresponding main object, which can make the cropped multiple candidate video cover frames highlight the main object, thereby attracting user attention and improving user click-through rate of the target video.

[0031] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0032] Figure 2 is a flow chart of a method for generating a video cover provided by an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps.

[0033] S210, performing first image feature extraction on multiple video frames of the target video respectively to obtain corresponding multiple first image features; and performing text feature extraction on video text information of the target video to obtain corresponding text features.

[0034] It should be noted that in the embodiments of the present application, the video cover generation method can be applied to a video cover generation device, or an electronic device integrated with the device. The electronic device can be implemented in various forms. For example, the electronic device can be a cloud service end (such as cloud service end 102), including but not limited to distributed storage system devices, server clusters, computing cloud service end clusters, etc.; for another example, the electronic device can be a user device (such as user device 106), including mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, etc., and the embodiments of the present application do not limit this.

[0035] It should also be noted that each video frame corresponds to a respective first image feature, and each first image feature can characterize information such as texture, color, structure, shape, grayscale, brightness contained in the corresponding video frame, and image correlation between the video frame and other video frames.

[0036] It should also be noted that the video text information of the target video may include at least one of the following: video title, video introduction and video tag.

[0037] In some embodiments, first image feature extraction is performed on multiple video frames of a target video to obtain corresponding multiple first image features; and text feature extraction is performed on video text information of the target video to obtain corresponding text features, which may include: first image feature extraction is performed on multiple video frames to obtain corresponding multiple first image features through a contrastive language-image pretraining (CLIP) model; and text feature extraction is performed on video text information to obtain corresponding text features.

[0038] It should be noted that the CLIP model is a multimodal pre-trained neural network model that uses a large amount of paired data of images and texts for pre-training to learn the alignment relationship between images and texts. The CLIP model consists of two parts: an image encoder (such as a Vision Transformer (VIT) encoder) and a text encoder (such as a Bidirectional Encoder Representations from Transformers (BERT) encoder). The image encoder can convert images into image features; the text encoder is responsible for converting text information into text features. The two encoders achieve cross-modal information interaction and fusion by sharing a vector space.

[0039] Furthermore, multiple video frames can be input into the image encoder in the CLIP model, so that the image encoder can perform first image feature extraction on the multiple video frames respectively to obtain multiple first image features; video text information can be input into the text encoder in the CLIP model, so that the text encoder can perform text feature extraction on the video text information to obtain text features.

[0040] Since the CLIP model is a multimodal pre-trained neural network model, and the CLIP model is trained based on a large number of sample images, compared with the methods in the related art, the multiple first image features and text features obtained by the CLIP model have higher accuracy, making the multiple first image features and text features more accurate, which is conducive to the subsequent selection of more accurate multiple candidate video cover frames based on the multiple first image features and text features.

[0041] In some embodiments, performing first image feature extraction on multiple video frames of the target video respectively may include: obtaining multiple original video frames of the target video, determining the multiple original video frames as multiple video frames in this step, and performing first image feature extraction on them.

[0042] It should be understood that, considering the large number of multiple original video frames of the target video, directly performing first image feature extraction on multiple original video frames will increase the complexity of feature extraction. Therefore, some video frames can be extracted from the target video, and the complexity of feature extraction can be reduced by performing first image feature extraction on the extracted video frames.

[0043] Further, in some embodiments, extracting multiple video frames from the target video may include: extracting frames of the target video based on a preset frame rate to obtain multiple video frames; or extracting multiple key frames from the target video and determining the multiple key frames as multiple video frames.

[0044] It should be noted that the preset frame rate can be set manually or in other ways, for example, extracting a frame from the target video at a certain interval, etc., and the embodiments of the present application are not limited to this.

[0045] It should also be noted that the multiple key frames are representative frames in the multiple original video frames. The multiple key frames are generally considered to be frames that contain important actions or changes in the multiple original video frames, and retain the key information of the multiple original video frames.

[0046] Through this method, multiple original video frames are extracted based on a preset frame rate, which can simplify the frame extraction process, and first image features are extracted based on the multiple video frames obtained by frame extraction, which can reduce the complexity of feature extraction; by using multiple key frames extracted from multiple original video frames as multiple video frames, the multiple video frames can retain the key information of the original video frames, and at the same time, first image features are extracted based on the multiple video frames, which can also reduce the complexity of feature extraction.

[0047] S220: Select multiple candidate video cover frames from multiple video frames based on the similarities between the multiple first image features and the text features.

[0048] It should be noted that the similarities between the plurality of first image features and the text features can be obtained through the CLIP model.

[0049] In the related art, there are two ways to obtain multiple candidate video cover frames. One is to extract multiple candidate video cover frames through a convolutional neural network. The quality of the multiple candidate video cover frames extracted by this method may be poor. The other is to determine whether multiple video frames meet pre-designed saturation, brightness and other conditions in turn, and determine the video frames that meet the conditions as multiple candidate video cover frames. The quality of the frames selected in this way is better than the first way, but may not fit the text features.

[0050] In the embodiment of the present application, the CLIP model can be used to more accurately extract multiple first image features and text features, and calculate the similarities between the multiple first image features and the text features, so as to more accurately find multiple candidate video cover frames with higher similarity.

[0051] In some embodiments, based on the similarities between multiple first image features and text features, selecting multiple candidate video cover frames from multiple video frames may include: taking video frames corresponding to a preset number of first image features with higher similarities as multiple candidate video cover frames.

[0052] The preset number may be set manually or in other ways, and this is not limited in the embodiments of the present application.

[0053] Exemplarily, assuming that the preset number is 10, video frames corresponding to the top 10 first image features with the highest similarity can be used as multiple candidate video cover frames.

[0054] Through this method, since the selection of multiple candidate video cover frames utilizes the similarity between multiple first image features and text features, the multiple candidate video cover frames can be closely related to the content of the target video.

[0055] It should be noted that, since the multiple candidate video cover frames are selected from the multiple video frames based on the similarities between the multiple first image features and the text features, and the text features usually include features related to the subject object, the multiple candidate video cover frames usually include at least one corresponding subject object. That is, usually, the preset number of video frames with the highest similarity between the first image features and the text features can be directly used as the multiple candidate video cover frames for subject object detection in step S230.

[0056] Exemplarily, video frames not containing the main object may be excluded from a preset number of video frames having the highest similarity between the multiple first image features and the text features, and the remaining video frames may be used as multiple candidate video cover frames to perform the main object detection in step S230.

[0057] S230: Detect at least one main object corresponding to each of the plurality of candidate video cover frames.

[0058] The so-called main object can be considered as the main object or focus object among the various objects contained in the candidate video cover frame. It is the object that occupies a main position or semantics, or a visual focus, among all the objects contained in the video frame. Since the first image feature of the candidate video cover frame matches the text feature, and the text feature usually indicates the main object in the video, for example, objects with people as the main body usually involve character descriptions in video titles and introductions, and objects with scenery as the main body usually involve scenery descriptions in video titles and introductions, therefore, the objects contained in the candidate video cover frame are usually the main objects in the video frame. The objects identified in the candidate video cover frame can be directly used as the main objects. Furthermore, related technologies can also be used to extract the main objects from the objects contained in the candidate video cover frame, exclude non-main objects, so that the obtained main objects can better reflect the main content of the video.

[0059] In some embodiments, detecting at least one main object corresponding to each of multiple candidate video cover frames may include: extracting second image features from each of the multiple candidate video cover frames to obtain corresponding multiple second image features; identifying the main object based on the multiple second image features, and determining at least one main object corresponding to each of the multiple candidate video cover frames based on the identification results.

[0060] Exemplarily, at least one main object corresponding to each of a plurality of candidate video cover frames may be detected by using an all-things detection model (such as a Yolo-World model).

[0061] In another feasible manner, a region of interest (ROI) detection may be performed on the candidate video cover frame, and the detected ROI includes at least one main object.

[0062] The second image feature may include an image feature of a candidate video cover frame within a detection frame area of ​​the subject object, or an image feature of a candidate video cover frame in the ROI.

[0063] It should be noted that the above-mentioned main objects include but are not limited to human portraits, or non-human portraits such as buildings, animals and plants, etc. In related technologies, the detection model can support the detection of one or more preset types of main objects.

[0064] By detecting at least one main object corresponding to each of multiple candidate video cover frames, it is convenient to subsequently crop the multiple candidate video cover frames based on the at least one main object, so that the cropped multiple candidate video cover frames highlight the main object, so as to attract user attention and increase the user's click rate on the target video.

[0065] It can be understood that, in the case where the main object includes a portrait or a non-portrait main object, the main object is identified based on multiple second image features, and based on the identification result, at least one main object corresponding to each of multiple candidate video cover frames is determined. There may be at least one of the following possible implementation methods.

[0066] In one possible implementation, portrait recognition can be performed based on multiple second image features to obtain at least one portrait corresponding to each second image feature; based on the at least one portrait corresponding to each second image feature, corresponding subject objects can be determined for multiple candidate video cover frames.

[0067] It should be noted that at least one human portrait corresponding to each second image feature may be used as at least one main object corresponding to each candidate video cover frame.

[0068] Through this method, by detecting at least one portrait corresponding to each of multiple candidate video cover frames, it is convenient to subsequently crop multiple candidate video cover frames based on the at least one portrait, so that the cropped multiple candidate video cover frames highlight the portrait, so as to attract user attention and improve the user's click rate on the target video.

[0069] Another possible implementation method is to perform subject object recognition based on multiple second image features, and obtain at least one non-portrait subject object corresponding to each second image feature; based on at least one non-portrait subject object corresponding to each second image feature, corresponding subject objects are determined for multiple candidate video cover frames respectively.

[0070] Through this method, by detecting at least one non-portrait main object corresponding to multiple candidate video cover frames, it is beneficial to subsequently crop multiple candidate video cover frames based on the at least one non-portrait main object, so that the cropped multiple candidate video cover frames highlight the main object to attract user attention and increase the user's click-through rate on the target video.

[0071] The main objects in the same candidate video cover frame may include only human portraits, only non-human portraits, or both human portraits and non-human portraits.

[0072] S240: Crop multiple candidate video cover frames based on at least one main object to generate multiple cropped candidate video cover frames, wherein the multiple cropped candidate video cover frames respectively contain the corresponding at least one main object.

[0073] It should be noted that, when the multiple candidate video cover frames after cropping respectively contain at least one corresponding main object, it is possible to avoid partial cropping of any main object respectively contained in the multiple candidate video cover frames, thereby ensuring that the main objects respectively contained in the multiple candidate video cover frames after cropping are complete main objects.

[0074] It should also be noted that when multiple candidate video cover frames are cropped based on at least one main object, the multiple candidate video cover frames after cropping can each contain as few areas as possible other than at least one main object, on the premise that the multiple candidate video cover frames each contain the corresponding at least one main object. This is conducive to blurring the background and highlighting the main object, so as to attract user attention and increase the user's click-through rate on the target video.

[0075] In some embodiments, cropping multiple candidate video cover frames based on at least one main object to generate multiple cropped candidate video cover frames may include: for each of the multiple candidate video cover frames, if there is a main object in the candidate video cover frame at least part of which is located in an edge area of ​​the candidate video cover frame, then cropping the main object at least part of which is located in the edge area from the candidate video cover frame.

[0076] It should be noted that the at least partial area of ​​the main object may be a partial area of ​​the main object or the entire area of ​​the main object, which is not limited in the embodiments of the present application.

[0077] It can be understood that in order to highlight the main object of the target video cover frame, a subject detection model is also introduced. Multiple candidate video cover frames are cropped according to at least one main object corresponding to each of the multiple candidate video cover frames, thereby improving the aesthetics of the final target video cover frame to attract users.

[0078] By using this method, the main object whose at least part of the area is located in the edge area of ​​the candidate video cover frame can be considered as an object that the user is not easily interested in or an object in the background. By cutting out the main object whose at least part of the area is located in the edge area from the candidate video cover frame, the main object that the user is not easily interested in can be cut out, so that the cropped multiple candidate video cover frames highlight the main object that the user is more likely to be interested in, thereby attracting more user attention and increasing the user's click rate on the target video.

[0079] S250: Determine a target video cover frame of a target video from the cropped multiple candidate video cover frames.

[0080] It should be noted that the target video cover frame is one of the cropped candidate video cover frames.

[0081] It can be understood that the target video cover frame is closely related to the content of the target video and can highlight the main object to attract the user's attention. However, without further processing the cropped multiple candidate video cover frames, there may be frames with poor aesthetics (such as blurred frames or frames with blocked areas) among the cropped multiple candidate video cover frames, thereby affecting the aesthetics of the target video cover frame.

[0082] Based on this, in some embodiments, determining the target video cover frame of the target video from the cropped multiple candidate video cover frames may include: performing image filtering processing and / or image enhancement processing on the cropped multiple candidate video cover frames to obtain multiple processed candidate video cover frames; determining the target video cover frame from the processed multiple candidate video cover frames.

[0083] Through this method, by performing image filtering and / or image enhancement processing on the cropped multiple candidate video cover frames, the aesthetics of the processed multiple candidate video cover frames can be improved, which is conducive to obtaining a target video cover frame whose aesthetics are more in line with the public aesthetics.

[0084] It can be understood that the image filtering process is used to screen out multiple candidate video cover frames with valid image areas; the image enhancement process is used to make the enhanced multiple candidate video cover frames have a higher quality presentation effect.

[0085] Furthermore, in some embodiments, image filtering processing is performed on the cropped multiple candidate video cover frames, which may include at least one of the following: removing text in the cropped multiple candidate video cover frames; cropping the image-free areas in the cropped multiple candidate video cover frames; filtering the cropped candidate video cover frames containing occluded areas.

[0086] It should be noted that removing text from the cropped multiple candidate video cover frames may include: filtering out text from the cropped multiple candidate video cover frames; or cropping regions containing text from the cropped multiple candidate video cover frames.

[0087] It should be understood that filtering out the text in the cropped multiple candidate video cover frames to obtain the processed multiple candidate video cover frames can be understood as that the processed multiple candidate video cover frames do not contain text, and the area where the processed multiple candidate video cover frames are located is the same as the area where the cropped multiple candidate video cover frames are located.

[0088] It should also be understood that the text in the cropped multiple candidate video cover frames is generally considered to be redundant text (i.e., unnecessary / illegal / non-compliant text), and therefore it is necessary to remove the text in the cropped multiple candidate video cover frames to improve the aesthetics of the processed multiple candidate video cover frames.

[0089] Exemplarily, an optical character recognition (OCR) algorithm may be used to recognize text in the cropped multiple candidate video cover frames, so as to remove the text in the cropped multiple candidate video cover frames.

[0090] For example, taking a cropped candidate video cover frame as an example, assuming that the candidate video cover frame includes the words "ABCDE", "ABCDE" can be filtered out from the candidate video cover frame, and the removed pixels can also be completed based on the surrounding pixels; or, when the lower edge of the candidate video cover frame is closest to the words "ABCDE", the area from 5 mm above the upper edge of the words "ABCDE" to the lower edge of the frame is determined as the area containing text, and the area containing text is cropped; or, when the upper edge of the candidate video cover frame is closest to the words "ABCDE", the area from 5 mm below the lower edge of the words "ABCDE" to the upper edge of the frame is determined as the area containing text, and the area containing text is cropped.

[0091] Exemplarily, the image-free area can be understood as an area without any valid image information, such as a black-bordered area or a white-bordered area. Furthermore, a detection algorithm can be used to detect whether there is an image-free area in the cropped multiple candidate video cover frames, so as to crop the image-free area. For example, a Canny detection algorithm can be used to obtain the edges of multiple candidate video cover frames. For the obtained Canny lines, the pixel mean on the left and right sides can be counted. If the pixel mean on one side tends to be black (such as a pixel value of 0), it is considered that there is an image-free area on this side.

[0092] Among them, effective image information refers to the pixel part in the image that has practical significance and value for the set task (such as object recognition task, etc.), and can also be understood as the pixel part with semantic meaning.

[0093] Exemplarily, the occluded area can be understood as an image area occluded by any mark, such as a mosaic area, etc. Further, a detection algorithm can be used to detect whether the cropped multiple candidate video cover frames contain the occluded area, and the frames containing the occluded area are discarded.

[0094] Through this method, image filtering processing is performed on the cropped multiple candidate video cover frames, which can improve the effectiveness and aesthetics of the processed multiple candidate video cover frames, thereby facilitating obtaining a target video cover frame that contains effective image information and whose aesthetics are more in line with the public's aesthetics.

[0095] Further, in some embodiments, performing image enhancement processing on the cropped multiple candidate video cover frames may include at least one of the following: deblurring the cropped multiple candidate video cover frames; and enhancing the brightness of the cropped multiple candidate video cover frames.

[0096] For example, a blur detection algorithm (such as a Linearity Image Quality Assessment (Linearity-IQA) algorithm) can be used to detect whether there are blurry frames in the cropped multiple candidate video cover frames, and a super-resolution algorithm (such as a Super-Resolution Generative Adversarial Network (SRGAN) algorithm) can be used to deblur the blurry frames. However, it is not limited to this, and other deblurring methods are also applicable to the solutions of the embodiments of the present application.

[0097] Exemplarily, those skilled in the art may implement brightness enhancement of an image in any appropriate manner, such as using a histogram equalization algorithm to adaptively enhance the brightness of multiple cropped candidate video cover frames, and so on.

[0098] Through this method, by performing image enhancement processing on the cropped multiple candidate video cover frames, the aesthetics of the processed multiple candidate video cover frames can be improved, which is conducive to obtaining a target video cover frame whose aesthetics is more in line with the user's aesthetic taste.

[0099] It can be understood that the post-processing (such as image filtering and / or image enhancement processing) of the cropped multiple candidate video cover frames provided in the embodiment of the present application is significantly different from that in the related art.

[0100] The related technology directly selects the target video cover frame from the cropped multiple candidate video cover frames without further processing the cropped multiple candidate video cover frames, which can easily lead to problems such as blurring, poor character expressions, mosaics, black borders, etc. in the selected target video cover frame.

[0101] In the embodiment of the present application, the aesthetics of the target video cover frame is improved by post-processing the cropped multiple candidate video cover frames. For example, a black edge removal algorithm, a mosaic detection algorithm, and a deblurring algorithm are used to generate a cover image that is more aesthetically pleasing to the public.

[0102] It can be seen that through the video cover generation method provided by the embodiment of the present application, on the one hand, the selection of multiple candidate video cover frames utilizes the similarity between multiple first image features and text features, which can make the multiple candidate video cover frames closely related to the content of the target video; on the other hand, the cropped multiple candidate video cover frames respectively contain at least one corresponding main object, which can make the cropped multiple candidate video cover frames highlight the main object, thereby attracting user attention and improving user click-through rate of the target video.

[0103] In the embodiment of the present application, multiple candidate video cover frames can be cropped based on at least one main object to obtain multiple cropped candidate video cover frames. Figure 3 A method for cropping multiple candidate video cover frames is described exemplarily.

[0104] Figure 3 is a schematic diagram of the process of cropping candidate video cover frames in the above-mentioned video cover generation method, such as Figure 3 As shown, it may include the following steps:

[0105] S310: Calculate the distances between the centers of multiple candidate video cover frames and the corresponding at least one main object respectively.

[0106] S320: Using the main object closest to the centers of the multiple candidate video cover frames as the cropping center, cropping the multiple candidate video cover frames to obtain multiple cropped candidate video cover frames.

[0107] It should be understood that the distance between the center of the multiple candidate video cover frames and the corresponding at least one main object can be understood as the distance between the center of the candidate video cover frame and the center of the detection frame of the corresponding at least one main object for each frame in the multiple candidate video cover frames. For each candidate video cover frame, the main object closest to the center of the candidate video cover frame can be selected from the main objects corresponding to it as the cropping center, and the candidate video cover frame can be cropped.

[0108] In some embodiments, taking a candidate video cover frame as an example, taking the main object closest to the center of the candidate video cover frame as the cropping center, and cropping multiple candidate video cover frames can include: determining the detection frame center of the main object closest to the center of the candidate video cover frame, taking the detection frame center as the cropping center, and continuously expanding the cropping size outward until the frame obtained after cropping the candidate video cover frame can contain at least one main object.

[0109] For example, taking a candidate video cover frame as an example, assuming that there are three characters A, B, and C in the candidate video cover frame, among which character B is closest to the center of the candidate video cover frame, character B can be used as the cropping center to crop the candidate video cover frame.

[0110] Furthermore, taking character B as the cropping center and cropping the candidate video cover frame may include: determining the detection frame center of character B; taking the detection frame center of character B as the cropping center, and continuously expanding the cropping size outward until the frame obtained after cropping the candidate video cover frame can include characters A, B and C.

[0111] According to the method, since the center of multiple candidate video cover frames is usually the visual focus, the main object closest to the center of multiple candidate video cover frames is the main object that users are more likely to be interested in. Therefore, by using the main object closest to the center of multiple candidate video cover frames as the cropping center and cropping multiple candidate video cover frames, the cropped multiple candidate video cover frames can highlight the main object that users are more likely to be interested in, thereby attracting more user attention and improving the user's click rate on the target video.

[0112] In the embodiment of the present application, the target video cover frame of the target video can be determined from the cropped multiple candidate video cover frames. Figure 4 The method for determining the target video cover frame is elaborated in detail.

[0113] Figure 4 is a schematic diagram of an exemplary process of determining a target video cover frame from multiple candidate video cover frames in the above-mentioned video cover generation method, such as Figure 4 As shown, it may include the following steps:

[0114] S410, evaluating the appearance of the cropped multiple candidate video cover frames to see whether their appearances meet the cover standard, and obtaining an evaluation result;

[0115] S420: Based on the evaluation result, determine a target video cover frame from the cropped multiple candidate video cover frames.

[0116] For example, suppose there are 3 candidate video cover frames after cropping, and the 5 frames can be scored according to the three dimensions of clarity, hue, and contrast. The 3 frames are scored in sequence by clarity as 2 points, 5 points, and 8 points; the 3 frames are scored in sequence by hue as 3 points, 9 points, and 4 points; the 3 frames are scored in sequence by contrast as 4 points, 7 points, and 6 points; the average of the scores of these three dimensions is 3 points, 7 points, and 6 points; the frame with the highest score of 7 points is selected from the average scores of these 3 frames as the target video cover frame.

[0117] It should be noted that, for the cropped candidate video cover frame that meets the cover standard, it means that the cropped candidate video cover frame has a higher aesthetics and is more able to attract user attention.

[0118] It should also be noted that, when image filtering and / or image enhancement processing is performed on the cropped multiple candidate video cover frames, an appearance evaluation can be performed on the processed multiple candidate video cover frames to determine whether their appearance meets the cover standards, and based on the evaluation results, a target video cover frame can be determined from the processed multiple candidate video cover frames.

[0119] Through this method, by evaluating whether the appearance of multiple cropped candidate video cover frames meets the cover standard, frames with better evaluation results can be screened out from the multiple cropped candidate video cover frames as target video cover frames, thereby making the target video cover frames more beautiful and more able to attract user attention.

[0120] It can be understood that the appearance evaluation of the cropped multiple candidate video cover frames to determine whether the appearance meets the cover standard may be implemented in at least one of the following possible ways.

[0121] In one possible implementation, the appearance of the cropped multiple candidate video cover frames can be evaluated to see whether they meet the cover standards from at least one of the following dimensions: clarity, hue, contrast, brightness, whether text exists, and whether the main object is highlighted.

[0122] Exemplarily, a scoring system may be adopted to evaluate the appearance of the cropped multiple candidate video cover frames. For example, the scores of the cropped multiple candidate video cover frames are sorted from high to low, and the aesthetics of the corresponding cropped multiple candidate video cover frames are sorted from high to low. At this time, the cropped candidate video cover frame with the highest score may be used as the target video cover frame. The score of the cropped candidate video cover frame may be obtained by scoring the cropped candidate video cover frames in at least one dimension and then taking the average value.

[0123] Exemplarily, the appearance evaluation of the cropped multiple candidate video cover frames can be performed in text form. For example, the cropped multiple candidate video cover frames are evaluated as "excellent", "good", "medium", and "poor" from at least one dimension, and the cropped candidate video cover frame with the most "excellent" is obtained as the target video cover frame.

[0124] Through this method, the appearance of multiple cropped candidate video cover frames is evaluated to see whether they meet the cover standard through at least one dimension, which can improve the efficiency of obtaining the target video cover frame and make the target video cover frame more beautiful and more able to attract user attention.

[0125] Another possible implementation method is to use a machine learning model for evaluating whether the appearance of an image meets the cover standard to evaluate the appearance of multiple cropped candidate video cover frames to see whether they meet the cover standard.

[0126] In some embodiments, the cropped multiple candidate video cover frames can be input into a machine learning model for evaluating whether the image appearance meets the cover standard to perform appearance evaluation on the cropped multiple candidate video cover frames.

[0127] It should be noted that the machine learning model can be a trained model.

[0128] Exemplarily, the machine learning model can be an aesthetic scoring model. The machine learning model can be a residual network (Residual Network, ResNet) model, or a transformer (Transformer) model with higher accuracy, which is not limited in the embodiments of the present application.

[0129] Exemplarily, the machine learning model may output scores of each of the plurality of cropped candidate video cover frames, so that the cropped candidate video cover frame with the highest score may be used as the target video cover frame.

[0130] In some embodiments, a sample image and a score label of the sample image can be obtained. The score label is used to indicate the appearance score of the sample image, such as an aesthetic score, etc. The score label can be obtained by taking the average value after at least one user scores the appearance of the sample image; the sample image is input into the machine learning model to be trained to evaluate whether the appearance of the sample image meets the cover standard, and the predicted score of the sample image is obtained; based on the predicted score, the score label and the preset loss function, the corresponding loss value is obtained; based on the loss value, the model parameters of the machine learning model to be trained are adjusted until the training termination condition is reached, thereby obtaining a trained machine learning model. Among them, the loss function can be implemented by a technician in this field according to the form of the specific machine learning model adopted, and the embodiment of the present application does not limit this.

[0131] It should be noted that the training termination condition may be that the training reaches a preset number of times, or the loss value meets a preset threshold value. Among them, the preset number of times and the preset threshold value can be appropriately set by those skilled in the art according to actual conditions, and the embodiments of the present application are not limited to this.

[0132] By using a machine learning model for evaluating whether the appearance of an image meets the cover standards, an appearance evaluation is performed on multiple cropped candidate video cover frames to determine whether the appearance meets the cover standards, which can make the aesthetics evaluation of the target video cover frame more accurate and objective.

[0133] Furthermore, after obtaining the target video cover frame of the target video, the target video cover frame may be scaled and adjusted according to a preset size to obtain an adjusted target video cover frame as the final target video cover frame.

[0134] It should be noted that by scaling the target video cover frame, the adjusted target video cover frame can be made to better meet the preset size standards and be more consistent with the specifications of other images in the interface where it is displayed, thereby improving the overall neatness and aesthetics of the entire interface.

[0135] It can be understood that in order to meet the business / user requirements for the size of the target video cover frame, the embodiment of the present application also introduces an adaptive cropping algorithm (such as a Resize function), which will be used to adaptively expand the length and width of the subject detection frame and finally generate a new frame.

[0136] Exemplarily, the target video cover frame can be scaled and adjusted by the Resize function. The Resize function can be used to adjust the size of the target video cover frame, that is, to change the width and height of the target video cover frame. For example, the Resize function can receive the target video cover frame and a preset size standard as input parameters, and output the resized target video cover frame.

[0137] It should be noted that, for a user device, the user device can send a target video to a cloud server, and the cloud server is used to obtain a target video cover frame of the target video (the target video cover frame may be an adjusted frame) through the video cover generation method provided in an embodiment of the present application; receive the target video cover frame sent by the cloud server; and display the target video cover frame at a preset display position corresponding to the target video, so as to enable the target user to recommend the target video to other users.

[0138] When the video cover generation method provided in the embodiment of the present application is applied to a user device, after the target video cover frame of the target video is obtained by the user device, the target video cover frame can be scaled and adjusted to obtain an adjusted target video cover frame; the adjusted target video cover frame is displayed at a preset display position corresponding to the target video to recommend the target video. The target video obtained by the user device can be uploaded by the target user or directly obtained by the user device, which is not limited in the embodiment of the present application.

[0139] The video cover generation method provided in the above embodiment is described in detail below in combination with specific application scenarios.

[0140] Figure 5 is a detailed flow chart of a video cover generation method provided by an embodiment of the present application, such as Figure 5 As shown, the method may include the following steps:

[0141] S501: Obtain target video.

[0142] S502: Extract multiple original video frames of the target video to obtain multiple video frames.

[0143] It should be noted that the plurality of original video frames may be decimated based on a preset frame decimation rate, thereby simplifying the frame decimation process.

[0144] S503, performing first image feature extraction on multiple video frames respectively to obtain corresponding multiple first image features; and performing text feature extraction on video text information of the target video to obtain corresponding text features.

[0145] Exemplarily, multiple video frames can be input into the VIT encoder in the CLIP model, so that the VIT encoder can perform first image feature extraction on the multiple video frames respectively to obtain multiple first image features; video text information can be input into the BERT encoder in the CLIP model, so that the BERT encoder can perform text feature extraction on the video text information to obtain text features.

[0146] S504: Select multiple candidate video cover frames from multiple video frames based on the similarities between the multiple first image features and the text features.

[0147] It should be noted that the video frames corresponding to a preset number of first image features with relatively high similarity may be used as a plurality of candidate video cover frames.

[0148] S505: Detect at least one main object corresponding to each of the plurality of candidate video cover frames.

[0149] Exemplarily, at least one main object corresponding to each of a plurality of candidate video cover frames may be detected by a Yolo-World model.

[0150] S506: Identify text in multiple candidate video cover frames.

[0151] For example, an OCR algorithm may be used to identify text in the cropped multiple candidate video cover frames.

[0152] S507: Crop multiple candidate video cover frames based on at least one main object to generate multiple cropped candidate video cover frames, and remove text from the cropped multiple candidate video cover frames.

[0153] It should be noted that the cropped multiple candidate video cover frames respectively contain at least one corresponding main object.

[0154] It should also be noted that removing text from the cropped multiple candidate video cover frames may include: filtering out text from the cropped multiple candidate video cover frames; or cropping regions containing text from the cropped multiple candidate video cover frames.

[0155] S508: Perform image filtering processing on the cropped multiple candidate video cover frames.

[0156] It should be noted that the image filtering processing of the cropped multiple candidate video cover frames may include: cropping the image-free areas in the cropped multiple candidate video cover frames; and / or filtering the cropped candidate video cover frames containing occluded areas.

[0157] S509: Perform image enhancement processing on the multiple candidate video cover frames after filtering.

[0158] It should be noted that performing image enhancement processing on the filtered multiple candidate video cover frames may include: deblurring the filtered multiple candidate video cover frames; and / or, enhancing the brightness of the filtered multiple candidate video cover frames.

[0159] S510: Determine a target video cover frame from a plurality of candidate video cover frames after image enhancement processing.

[0160] It should be noted that an appearance evaluation may be performed on the multiple candidate video cover frames after image enhancement to determine whether their appearance meets the cover standard, and based on the evaluation result, a target video cover frame may be determined from the multiple candidate video cover frames after image enhancement.

[0161] For example, the appearance of multiple candidate video cover frames after image enhancement processing can be evaluated from at least one of the following dimensions to see whether their appearance meets the cover standard: clarity, hue, contrast, brightness, whether text exists, and whether the main object is highlighted. For example, the candidate video cover frames after image enhancement processing can be scored from at least one dimension and then averaged, and the frame with the highest score is used as the target video cover frame.

[0162] Exemplarily, a machine learning model for evaluating whether the appearance of an image meets the cover standard can be used to evaluate the appearance of multiple candidate video cover frames after image enhancement processing to see whether their appearance meets the cover standard. For example, the machine learning model can output the scores of the multiple candidate video cover frames after image enhancement processing, and the frame with the highest score is used as the target video cover frame.

[0163] An embodiment of the present application provides a method for generating a video cover. On the one hand, the selection of multiple candidate video cover frames utilizes the similarity between multiple first image features and text features, which can make the multiple candidate video cover frames closely related to the content of the target video; on the other hand, the cropped multiple candidate video cover frames respectively contain at least one corresponding main object, which can make the cropped multiple candidate video cover frames highlight the main object, thereby attracting user attention and improving user click-through rate of the target video.

[0164] For example, Figure 6 Schematic diagram of a scenario for determining a short video cover provided by an embodiment of the present application. Figure 6As shown, suppose there is a short video representing the friendship of three persons x, y, and z. Frames of the short video are extracted to obtain extracted frames a, b, c, d, and e. Image features are extracted from frames a, b, c, d, and e to obtain corresponding image features a1, b1, c1, d1, and e1. Text features are extracted from the video text information of the short video to obtain the corresponding text feature f. From the image features a1, b1, c1, d1, and e1, the frames a, b1, and e1 corresponding to the image feature a1 having the highest similarity to the text feature f are selected. The invention relates to frame c corresponding to image feature c1 and frame d corresponding to image feature d1; detect the portraits of persons x, y, and z corresponding to frames a, c, and d respectively, and crop frames a, c, and d based on the portraits of persons x, y, and z, so that the cropped frames a, c, and d include the portraits of persons x, y, and z; score the cropped frames a, c, and d respectively from the dimension of whether the main object is highlighted, and obtain the scores of cropped frames a, c, and d being 9, 7, and 5 respectively, so that the cropped frame a with the highest score can be used as the video cover of the short video.

[0165] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the idea of ​​the present application, they should also be regarded as the contents disclosed in the present application. For another example, under the premise of no conflict, the various embodiments and / or the technical features in the various embodiments described in the present application can be arbitrarily combined with the prior art, and the technical solution obtained after the combination should also fall within the protection scope of the present application.

[0166] It can be understood that in the various method embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0167] Based on the same inventive concept as the above embodiments, Figure 7 is a schematic diagram of the structure of a video cover generation device provided in an embodiment of the present application, such as Figure 7As shown, the video cover generation device 600 may include a feature extraction module 610, a selection module 620, a detection module 630, a cropping module 640, and a determination module 650, wherein:

[0168] The feature extraction module 610 is used to extract first image features from multiple video frames of the target video respectively to obtain corresponding multiple first image features; and to extract text features from video text information of the target video to obtain corresponding text features;

[0169] A selection module 620, configured to select a plurality of candidate video cover frames from a plurality of video frames based on the similarities between the plurality of first image features and the text features respectively;

[0170] A detection module 630, configured to detect at least one subject object corresponding to each of the plurality of candidate video cover frames;

[0171] A cropping module 640, configured to crop a plurality of candidate video cover frames based on at least one subject object to generate a plurality of cropped candidate video cover frames, wherein the plurality of cropped candidate video cover frames respectively contain at least one corresponding subject object;

[0172] The determination module 650 is used to determine a target video cover frame of a target video from the cropped multiple candidate video cover frames.

[0173] In some embodiments, Figure 7 As shown, the video cover generation device 600 may further include an identification module 660, wherein:

[0174] The feature extraction module 610 is further used to extract second image features from the plurality of candidate video cover frames respectively to obtain corresponding plurality of second image features;

[0175] The recognition module 660 is used to perform subject object recognition based on the plurality of second image features to obtain a recognition result;

[0176] The determination module 650 is used to determine at least one main object corresponding to each of the plurality of candidate video cover frames based on the recognition result.

[0177] In some embodiments, the recognition module 660 is further used to perform portrait recognition based on multiple second image features to obtain at least one portrait corresponding to each second image feature; the determination module 650 is further used to determine the corresponding subject objects for multiple candidate video cover frames based on at least one portrait corresponding to each second image feature.

[0178] In some embodiments, the cropping module 640 is further used to respectively calculate the distance between the centers of multiple candidate video cover frames and the corresponding at least one main object; the main object closest to the centers of the multiple candidate video cover frames is used as the cropping center to crop the multiple candidate video cover frames.

[0179] In some embodiments, the cropping module 640 is also used to, for each frame of a plurality of candidate video cover frames, if there is a main object in the candidate video cover frame at least part of which is located in the edge area of ​​the candidate video cover frame, then the main object at least part of which is located in the edge area is cropped from the candidate video cover frame.

[0180] In some embodiments, Figure 7 As shown, the video cover generation device 600 may further include a processing module 670, wherein:

[0181] The processing module 670 is used to perform image filtering processing and / or image enhancement processing on the cropped multiple candidate video cover frames to obtain multiple processed candidate video cover frames;

[0182] The determination module 650 is further configured to determine a target video cover frame from the processed multiple candidate video cover frames.

[0183] In some embodiments, the processing module 670 is also used for one of the following: removing text from the cropped multiple candidate video cover frames; cropping the image-free areas in the cropped multiple candidate video cover frames; filtering the cropped candidate video cover frames containing occluded areas.

[0184] In some embodiments, the processing module 670 is further used for one of the following: deblurring the cropped multiple candidate video cover frames; and enhancing the brightness of the cropped multiple candidate video cover frames.

[0185] In some embodiments, Figure 7 As shown, the video cover generation device 600 may further include an evaluation module 680, wherein:

[0186] An evaluation module 680 is used to evaluate the appearance of the cropped multiple candidate video cover frames to determine whether the appearance meets the cover standard, and obtain an evaluation result;

[0187] The determination module 650 is further configured to determine a target video cover frame from the cropped multiple candidate video cover frames based on the evaluation result.

[0188] In some embodiments, the evaluation module 680 is further used to input the cropped multiple candidate video cover frames into a machine learning model for evaluating whether the image appearance meets the cover standard, so as to perform appearance evaluation on the cropped multiple candidate video cover frames.

[0189] In some embodiments, the feature extraction module 610 is further used to extract first image features from multiple video frames respectively by comparing the language-image pre-trained CLIP model to obtain corresponding multiple first image features; and to extract text features from the video text information to obtain corresponding text features; and to obtain the similarity between the multiple first image features and the text features respectively through the CLIP model.

[0190] In the scheme of the embodiment of the present application, on the one hand, the selection of multiple candidate video cover frames utilizes the similarity between multiple first image features and text features, which can make the multiple candidate video cover frames closely related to the content of the target video; on the other hand, the cropped multiple candidate video cover frames respectively contain at least one corresponding main object, which can make the cropped multiple candidate video cover frames highlight the main object, thereby attracting user attention and improving user click-through rate on the target video.

[0191] Those skilled in the art should understand that the relevant description of the above-mentioned video cover generation device in the embodiment of the present application can be understood by referring to the relevant description of the video cover generation method in the embodiment of the present application.

[0192] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0193] like Figure 8 As shown, the electronic device 700 may include: a processor (Processor) 710, a communication interface (Communications Interface) 720, a memory (Memory) 730, and a communication bus 740; wherein:

[0194] The processor 710 , the communication interface 720 , and the memory 730 communicate with each other via the communication bus 740 .

[0195] The communication interface 720 is used to communicate with other electronic devices or servers.

[0196] The processor 710 is used to execute the program 750, and specifically can execute the relevant steps in the above-mentioned video cover generation method embodiment.

[0197] Specifically, the program 750 may include program codes, which include computer operation instructions.

[0198] The processor 710 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0199] The memory 730 is used to store the program 750. The memory 730 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0200] The program 750 may include multiple computer instructions. Specifically, the program 750 may enable the processor 710 to execute operations corresponding to the video cover generation method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0201] The specific implementation of each step in program 750 can refer to the corresponding description of the corresponding steps and units in the above method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0202] The present application also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any of the above-mentioned multiple method embodiments is implemented. The computer storage medium includes but is not limited to: a compact disc read-only memory (CD-ROM), a random access memory (RAM), a floppy disk, a hard disk or a magneto-optical disk, etc.

[0203] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to the video cover generation method described in any one of the above-mentioned multiple method embodiments.

[0204] In some embodiments, the embodiments of the present application also provide a computer program, which, when executed on a computing device, enables the computing device to perform operations corresponding to the video cover generation method described in any one of the above-mentioned method embodiments.

[0205] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0206] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0207] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., a random access memory (RAM), a read-only memory (ROM), a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0208] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0209] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A method for generating a video cover, comprising: Extracting first image features from multiple video frames of the target video respectively to obtain corresponding multiple first image features; and extracting text features from video text information of the target video to obtain corresponding text features; Selecting a plurality of candidate video cover frames from the plurality of video frames based on the similarities between the plurality of first image features and the text features respectively; Detecting at least one subject object respectively corresponding to a plurality of the candidate video cover frames, and cropping the plurality of the candidate video cover frames based on the at least one subject object to generate a plurality of cropped candidate video cover frames, wherein the plurality of cropped candidate video cover frames respectively contain the corresponding at least one subject object; A target video cover frame of the target video is determined from the cropped multiple candidate video cover frames.

2. The method according to claim 1, wherein: The detecting at least one main object corresponding to each of the plurality of candidate video cover frames comprises: Extracting second image features from each of the plurality of candidate video cover frames to obtain corresponding plurality of second image features; The subject object is identified based on the plurality of the second image features, and based on the identification result, at least one subject object corresponding to each of the plurality of the candidate video cover frames is determined.

3. The method according to claim 2, wherein: The performing subject object recognition based on the plurality of second image features, and determining at least one subject object corresponding to each of the plurality of candidate video cover frames based on the recognition result, comprises: Performing portrait recognition based on the plurality of second image features to obtain at least one portrait corresponding to each second image feature; Based on at least one human portrait corresponding to each of the second image features, corresponding subject objects are respectively determined for the plurality of candidate video cover frames.

4. The method according to any one of claims 1 to 3, wherein: The step of cropping the plurality of candidate video cover frames based on at least one of the subject objects comprises: respectively calculating the distances between the centers of the plurality of candidate video cover frames and the corresponding at least one subject object; The main object closest to the centers of the multiple candidate video cover frames is used as a cropping center to crop the multiple candidate video cover frames.

5. The method according to any one of claims 1 to 3, wherein: The step of cropping the plurality of candidate video cover frames based on at least one of the subject objects to generate the cropped plurality of candidate video cover frames includes: For each of the multiple candidate video cover frames, if there is a main object in the candidate video cover frame, at least part of its area is located in the edge area of ​​the candidate video cover frame, then the main object at least part of its area is located in the edge area is cut out from the candidate video cover frame.

6. The method according to any one of claims 1 to 3, wherein: The step of determining a target video cover frame of the target video from the cropped plurality of candidate video cover frames comprises: Performing image filtering processing and / or image enhancement processing on the cropped multiple candidate video cover frames to obtain multiple processed candidate video cover frames; The target video cover frame is determined from the processed multiple candidate video cover frames.

7. The method according to claim 6, wherein: The performing image filtering processing on the cropped multiple candidate video cover frames includes at least one of the following: Removing text from the cropped multiple candidate video cover frames; Cropping the image-free areas in the cropped multiple candidate video cover frames; The cropped candidate video cover frame containing the blocked area is filtered out.

8. The method according to claim 6, wherein: The performing image enhancement processing on the cropped multiple candidate video cover frames comprises at least one of the following: Deblurring the cropped multiple candidate video cover frames; Brightness enhancement is performed on the cropped multiple candidate video cover frames.

9. The method according to any one of claims 1 to 3, wherein: The step of determining a target video cover frame of the target video from the cropped plurality of candidate video cover frames comprises: An appearance evaluation is performed on the cropped multiple candidate video cover frames to determine whether their appearances meet the cover standard, and based on the evaluation result, the target video cover frame is determined from the cropped multiple candidate video cover frames.

10. The method according to claim 9, wherein: The performing appearance evaluation on the cropped multiple candidate video cover frames to determine whether their appearances meet the cover standard comprises: The cropped multiple candidate video cover frames are input into a machine learning model for evaluating whether the image appearance meets the cover standard, so as to perform appearance evaluation on the cropped multiple candidate video cover frames.

11. The method according to any one of claims 1 to 3, wherein: The first image feature extraction is performed on a plurality of video frames of the target video to obtain a plurality of corresponding first image features; and the text feature extraction is performed on the video text information of the target video to obtain corresponding text features, including: By comparing the language-image pre-trained CLIP model, first image features are extracted from the plurality of video frames to obtain corresponding plurality of first image features; and text features are extracted from the video text information to obtain corresponding text features; Furthermore, the similarities between the plurality of first image features and the text features are obtained through the CLIP model.

12. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1-11.

13. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the method according to any one of claims 1 to 11.

Images