Video cover generation method and device

Abstract

The invention discloses a video cover generation method and device, and the method comprises the steps: filtering frame images in a target video through a preset filtering condition, obtaining candidate frame images, and calculating the video frame features of each candidate frame image; calculating text features of text description corresponding to the video frame features; a cross-modal interaction mode is introduced, and the semantic correlation degree of each candidate frame image and the text features is measured; calculating and determining the selection probability that each candidate frame image becomes a cover frame according to the correlation degree, and determining a high-quality video frame which is most related to the text description as a candidate cover based on a probability value; and in response to a cover style specified by a user, performing personalized processing on the candidate cover to generate a target cover. The device comprises a processor and a memory. According to the method and the device, the problems that the automatic generation effect of the current video cover is poor and the relevancy between the generated cover and the video content is poor are solved, the personalized video cover can be automatically generated for the video, the user experience is improved, and the video attraction is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a method and device for generating a video cover. Background Art

[0002] In recent years, with the popularization of mobile Internet and smart terminals and video shooting devices represented by mobile phones, the threshold for people to shoot and share videos has become lower and lower. This has greatly promoted the rapid development of short video platforms. According to statistics, many video social platforms have more than 1 billion active users and more than 20 billion video playbacks every day. However, for massive video data, many non-professional video creators usually do not have enough time or expertise to select attractive video cover frames. And most video websites generally use the first few frames of the video as the default cover when the user does not upload the cover. However, these default covers are usually of low quality, blurry or too dark, and cannot accurately reflect the main content of the video, thus affecting the click-through rate and popularity of the video. At the same time, the original video image frames often have the disadvantages of insufficient content attractiveness and poor image style popularity. Therefore, how to realize intelligent video cover generation is an important problem that needs to be solved urgently.

[0003] Traditional cover generation methods are mainly based on image frame analysis of video modality. They often start from the color, clarity, brightness, content richness and other aspects of the video frame to comprehensively evaluate the image aesthetic information of the frame. However, this method relies too much on manually designed evaluation indicators and cannot optimally adapt to complex and diverse video data. With the rapid development of deep learning, video frame generation solutions based on deep learning can also generate cover images for videos, but this technical solution does not comprehensively consider the role of text modality data in the video, and the generated images often do not conform to the theme expressed by the video content.

[0004] In the real short video platform scenario, people often pay attention to the video cover and the corresponding text description, as well as the personalized artistic style of the cover. If the cover is different from the description and does not have a popular style, it will greatly affect browsing interest and click tendency. Therefore, a video cover that is consistent with the video description and has high quality and personalized style can improve user experience and enhance the appeal of the video. Summary of the invention

[0005] The present invention provides a method and device for generating a video cover. The present invention solves the problems that the automatic generation of video covers is poor, the generated covers are poorly related to the video content, and lack novelty. The present invention can automatically generate a personalized video cover for a video, improve user experience and enhance video attractiveness. See the following description for details:

[0006] A method for generating a video cover, the method comprising:

[0007] Filter the frame images in the target video by using preset filtering conditions to obtain candidate frame images, calculate the video frame features of each candidate frame image; calculate the text features of the text description corresponding to the video frame features;

[0008] A cross-modal interaction method is introduced to measure the semantic relevance between each candidate frame image and the text features; the probability of each candidate frame image being selected as the cover frame is calculated based on the relevance, so as to determine the high-quality video frame that is most relevant to the text description as the candidate cover based on the probability value;

[0009] In response to the cover style specified by the user, the candidate covers are personalized to generate a target cover.

[0010] The filtering of the frame images in the target video by the preset filtering conditions is specifically as follows:

[0011] Filtering the frame image in the target video with respect to image brightness, clarity and color uniformity to obtain a first filtering result;

[0012] Based on the edge quality change rate of two adjacent frame images that meet the time sequence in the first filtering result, a transition frame image is determined, and the transition frame image is filtered out to obtain a candidate frame image.

[0013] The method of calculating and determining the probability of each candidate frame image becoming a cover frame according to the degree of relevance, so as to determine a high-quality video frame that is most relevant to the text description as a candidate cover based on the probability value, is specifically as follows:

[0014] Mapping text features and video frame features into the same semantic space; calculating the distance between the text features and the video frame features in the semantic space, and determining the similarity between the text features and the video frame features based on the distance;

[0015] The probability of each candidate frame image being selected as a candidate cover is determined based on similarity calculation; and the candidate frame image with the largest probability value is determined as the candidate cover.

[0016] The method further includes: constructing a target model for selecting candidate covers, so as to select candidate covers for the video based on the target model, wherein the loss function of the target model is:

[0017]

[0018] Among them, ||·|| F represents the sum of squares of pixel values, The candidate cover selected by the model, I truth The actual candidate cover.

[0019] The step of performing personalized processing on the candidate cover in response to the cover style specified by the user to generate the target cover is as follows:

[0020] Determine a specific word of the cover style specified by the user; map the specific word into a first feature vector based on a word vector model;

[0021] Mapping the text embedding features corresponding to the candidate frame image into a second feature vector based on a word vector model; processing the first feature vector and the second feature vector based on a text encoder to generate a text guide feature;

[0022] Based on the text guidance features and the text embedding features corresponding to the candidate covers, stylized text embedding information matching the cover style specified by the user is generated; based on the stylized text embedding information, the candidate covers are processed to obtain a target cover with the cover style specified by the user.

[0023] In a second aspect, a video cover generation device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0024] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0025] The beneficial effects of the technical solution provided by the present invention are:

[0026] 1. This method can refer to both video and text modal information at the same time to achieve automatic generation of cross-modal video covers;

[0027] 2. This method performs dual-branch feature extraction on the video and the corresponding text description in the video platform, including: video branch preprocessing and frame feature extraction, text branch feature extraction; then the semantic correlation between the candidate frame and the text is calculated through the cross-attention mechanism, and the probability value of becoming the cover is calculated through the correlation, and then the candidate frame image with the highest probability is output as the candidate cover of the video. The process of calculating and determining the candidate cover using the two modal information of video and text can ensure the selection efficiency and cover quality, and avoid the phenomenon that the cover is irrelevant to the video content or has poor image quality;

[0028] 3. In order to meet the personalized needs of users, the present invention can also realize personalized style transfer, so that the style features specified by the user can be transferred to the candidate cover, and then generate a target cover with the user-specified style. This not only meets the personalized needs of users, but also increases the fun and attractiveness of the cover, and has strong application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of a method for generating a video cover;

[0030] Figure 2 An application flow chart of the video cover generation method;

[0031] Figure 3 Another application flow chart of the method for generating a video cover;

[0032] Figure 4 A block diagram of a video cover generation device. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0034] Example 1

[0035] An embodiment of the present invention provides a method for generating a video cover, the method comprising the following steps:

[0036] S1: filtering the frame images in the target video by a preset filtering condition to obtain a candidate frame image, where the filtering condition is related to the image quality and display effect of the frame image;

[0037] For example: filter out the black and white frames at the beginning of the video; filter out the quickly flashing blurred images; or images with obviously insufficient brightness; narrow the selection range of candidate cover frames to reduce the subsequent calculation burden.

[0038] S2: Calculate the video frame features of each candidate frame image;

[0039] S3: Calculate the text features of the text description corresponding to the target video;

[0040] S4: Introduce a cross-modal interaction method to measure the semantic relevance between each candidate frame image and the description text; calculate and determine the probability of each candidate frame image becoming the cover frame based on the relevance, so as to determine the high-quality video frame that is most relevant to the text description as the candidate cover based on the probability value;

[0041] S5: In response to the cover style specified by the user, the candidate cover is personalized to generate a target cover.

[0042] After determining the candidate cover, the user can directly use it as the final video cover, or personalize and stylize it to generate a cover with a specified style. For example, the user inputs a selected reference image with a popular style, and then uses personalized image diffusion processing, that is, generating a diffusion image, to successfully migrate the video frame to generate a personalized video cover.

[0043] Example 2

[0044] The solution in Example 1 is further introduced below in combination with specific examples and calculation formulas, as described below for details:

[0045] The above step S1 filters the frame images in the target video based on the preset filtering conditions and obtains the candidate frame images, including:

[0046] 1) filtering the frame image in the target video with respect to image brightness, clarity and color uniformity to obtain a first filtering result;

[0047] 2) Determine the transition frame image based on the edge quality change rate of two adjacent frame images that meet the time sequence in the first filtering result, and filter out the transition frame image to obtain the candidate frame image.

[0048] Specifically, for current videos, directly processing all frame images will introduce a large amount of data and is very time-consuming, while sampling video frames at uniform time intervals for subsequent cover selection processing may miss important video frames, which is not conducive to cover selection. However, considering that there are low-quality frames and blurred transition frames in the video frames, these video frames cannot become cover frame candidates because they do not have aesthetic properties. Accordingly, the embodiment of the present invention eliminates unqualified frame images by pre-processing all frame images, and at the same time narrows the frame image processing range, thereby reducing the processing load and improving processing efficiency.

[0049] Specifically, in the above step 1), the frame image in the target video is filtered with respect to image brightness, clarity and color uniformity to obtain a first filtering result, including:

[0050] The relative brightness of each frame is determined based on the following formula:

[0051] Luminance(I rgb )=02126I r +0.7152I g +0.0722I b

[0052] Among them, I r , I g and I b Represents the red, green, and blue (RGB) channel pixel mean of the frame image, Luminance (I rgb ) represents the relative brightness of the frame image.

[0053] The clarity of each frame is determined based on the following calculation:

[0054]

[0055] Among them, I gray Represents the grayscale pixel value of the video frame image, Δ x I gray and Δ y I gray Respectively represent the gradient of the grayscale image in the horizontal and vertical directions, Sharpness (I gray ) indicates the clarity of the frame image.

[0056] The color uniformity of each frame of image is calculated based on the following formula:

[0057]

[0058] Among them, hist(I gray ) is the normalized brightness histogram of the frame image, and then the values ​​are sorted in descending order, and the cumulative distribution at the first 5% of the histogram values ​​is calculated to obtain the Uniformity (I gray ), the smaller the value, the more uniform the color distribution in the image, and vice versa.

[0059] The above three indicators (relative brightness of each frame image, clarity of each frame image, and color uniformity of each frame image) are sorted from large to small, and filtered by a preset ratio threshold, and the first 40% of the image frames that simultaneously meet the corresponding sorting results of the three indicators are retained as the first filtering results.

[0060] Since transition frames usually exist in the process of video scene conversion, the embodiment of the present invention adopts shot boundary detection to realize the recognition of transition frames, specifically, the edge change rate between adjacent frames arranged in time series in the first filtering result is detected to identify the boundary of the video scene, and the frame image located near the boundary is determined and discarded. That is, the frame image is a transition frame, which is a low-quality frame image and is not considered to be used as a cover, so it can be removed.

[0061] In the above step S2, after obtaining the candidate frame images, the video frame features of each candidate frame image are calculated, including:

[0062] Based on the image encoder, each candidate frame image is processed in combination with the self-attention mechanism to obtain the video frame features corresponding to each candidate frame image.

[0063] In the embodiment of the present invention, a network model is used to extract video frame features. For example, the CLIP model can be used. CLIP is a large-scale visual language pre-training model. Its image encoder uses the VisionTransformer neural network model to process image information through a self-attention mechanism. Unlike traditional convolutional neural networks, it uses a group of Transformer modules to extract image features, making the model have better generalization ability and scalability. In the embodiment of the present invention, for the candidate frame image set V = {I1, I2, ..., I n}, where n represents the number of video frames, and each candidate frame image I i , i∈[1,n] is input into the image encoder of CLIP, and its feature representation can be obtained as:

[0064]

[0065] in, represents the image encoder of CLIP, d is the length of the feature dimension.

[0066] Next, in the above step S3, the text features of the text description corresponding to the target video are calculated, which includes: processing the text description corresponding to the target video based on a text encoder to obtain the text features.

[0067] The network model in the previous embodiment is used, and the CLIP model is still used in the embodiment of the present invention to realize the extraction of text features. The text encoder of the CLIP model adopts the BERT language model. The BERT model uses the Transformer module to process text information, which can automatically encode and represent text, thereby achieving better semantic understanding and expression. Specifically, for the text description T corresponding to the target video V, it is input into the CLIP text encoder to obtain the feature representation as:

[0068] u=φ(T)

[0069] where φ(·) denotes the text encoder of CLIP.

[0070] Continue to combine Figure 2 As shown, after the video frame features and text features of the candidate frame image are determined, it is necessary to determine their relationship, that is, to perform a correlation evaluation. In the embodiment of the present invention, step S4 determines the relationship between the text features and each video frame feature, and determines the candidate cover in the candidate frame image based on the relationship, including:

[0071] 1) Map text features and video frame features into the same semantic space;

[0072] 2) Calculate the distance between the text features and the video frame features in the semantic space, and determine the similarity between the text features and the video frame features based on the distance;

[0073] 3) Determine the probability of each candidate frame image being selected as a candidate cover based on similarity calculation;

[0074] 4) The candidate frame image with the largest probability value is determined as the candidate cover.

[0075] When executing the above method, for the acquired video frame feature v i , i∈[1, n] and text feature u, the embodiment of the present invention uses cross attention interaction to measure the semantic relevance between each candidate frame image and text description. Specifically, the video frame mapping network and the text mapping network are used to map the two modal features of video frame features and text features to a common semantic space, namely:

[0076]

[0077] Among them, f visual (·) and f textual (·) are fully connected neural networks respectively.

[0078] Then, in this common semantic space, the video frame features and text features are semantically comparable. Therefore, the cosine distance can be used to measure the semantic similarity between the video frame features and the text features:

[0079]

[0080] Next, we can build a target model to assist in selecting candidate covers. This target model is mainly used for probability calculation and selects candidate covers based on probability values:

[0081] 5) Construct a target model for selecting candidate covers, so as to select candidate covers for the video based on the target model, wherein the loss function of the target model is:

[0082]

[0083] Among them, ||·|| F represents the sum of squares of pixel values, The candidate cover selected by the model, I truth The actual candidate cover.

[0084] Specifically, based on all {s i}, i∈[1,n], the probability of each video frame being selected as the cover can be obtained as:

[0085]

[0086] Among them, p i ∈[0, 1] represents the correlation of the i-th video frame feature with respect to other frame features and text features. λ is the scaling factor, which is used to assign a larger probability value to video frames with greater semantic correlation and compress the remaining video frames with less semantic correlation. Based on this, in the modeling process, the video frame with the largest corresponding probability value is selected as the candidate cover:

[0087]

[0088] Among them, j * Index subscript for the cover frame.

[0089] During the training of the target model, for each video, the predicted video frame can be minimized Video cover with annotations I truth The distance between the two (that is, the training data includes: training videos, related degrees, and annotated video covers) to make the model predict the correct cover frame as accurately as possible. Therefore, the training loss function of the model is set to:

[0090]

[0091] Among them, ||·|| F Represents the sum of squares of pixel values. In practical applications, the training optimization process of the target model can use Adam as the optimizer, with the learning rate set to 1e-4 and the momentum parameter set to 0.9.

[0092] In real life, due to factors such as diverse content themes and varying shooting quality, videos do not completely possess popular styles that keep up with the times, which affects the click appeal of candidate covers to users.

[0093] To solve this problem, an embodiment of the present invention proposes a personalized style transfer intelligent generation method to transform the selected video candidate frame image into a cover image with a personalized popular style.

[0094] Specifically, step S5 in the above embodiment 1 performs personalized processing on the candidate cover in response to the cover style specified by the user to generate the target cover, including:

[0095] 1) Determine the specific words of the cover style specified by the user;

[0096] 2) Mapping the specific word into the first feature vector based on the word vector model;

[0097] 3) Mapping the text embedding features corresponding to the candidate frame image into a second feature vector based on the word vector model;

[0098] 4) generating a text guiding feature based on the text encoder processing the first feature vector and the second feature vector;

[0099] 5) generating stylized text embedding information matching the cover style specified by the user based on the text guidance features and the text embedding features corresponding to the candidate cover;

[0100] 6) Process candidate covers based on the stylized text embedding information to obtain a target cover with a user-specified cover style.

[0101] The step 6) of processing the candidate cover based on the stylized text embedding information to obtain the target cover with the cover style specified by the user includes:

[0102] (1) Train an initial image generation model based on specific words and images with the same style as the specific words;

[0103] (2) Determine historical candidate covers that have the same text embedding features as the candidate cover;

[0104] (3) training an initial image generation model based on the text guidance feature, the first feature vector, the text embedding feature of the historical candidate cover, and the historical candidate cover to obtain an image generation model that can match and process the candidate cover based on the text embedding feature to generate the target cover;

[0105] (4) The image generation model processes the candidate covers based on the stylized text embedding information to generate the target cover.

[0106] The specific path of generating a stylized target cover based on stylized text embedding information through an image generation model is as follows: Figure 3 As shown in Figure 1, personalized style transfer is divided into two steps, including: (1) learning the text semantic embedding features of the stylized image and the video candidate frame image respectively during the image generation process; (2) generating a stylized video cover under the text semantic prompts of the stylized video frame image.

[0107] Specifically, the user first selects an image style that meets personal preferences, and selects some images with this style as learning data, and trains the initial model accordingly, for example: based on a pre-trained generative model (in the embodiment of the present invention, the model is a latent diffusion model (LDM)), learn the text semantic embedding corresponding to this style. For example: a specific word S* is used to represent the image style selected by the user, and the image is input into the generative model for training. In this model, only the word vector model is learned by mapping specific words to word vectors through a vocabulary to optimize parameters, while the parameters of the remaining text encoders and image generators are kept fixed.

[0108] During the training process, a specific word S* is mapped to a feature vector s through the word vector model * , this vector is combined with other text segment feature vectors s i For example, the feature vector of the text embedding feature corresponding to the candidate frame image is converted into a text guide feature through the text encoder, and is input into the image generator to guide the diffusion model for image generation. Therefore, the loss function of the training is:

[0109] s * =argmin E z～ε(x)，τ，∈～N(0，1)，t [||∈-∈ θ (z t ,t,τ(S * ,S i ))|| 2 ]

[0110] Among them, ∈ is the sampling noise, ∈ θ is the noise predictor, which contains three inputs: the denoising result z of the previous step t , denoising step number t, text-guided feature τ(s * ,s i ).

[0111] Through this process, the stylized image features can be fine-tuned into the text embedding feature vector, that is, the stylized image features can be fine-tuned into the text embedding feature vector by learning s * To achieve stylized migration generation of other images, that is, to process frame images that do not have the style into images with the style, for example: to process the original frame image into ink style, oil painting style, comic style, etc., or a custom style, similar to a custom style "template" / "feature", and let the model learn the custom style to process the input frame image into an image with the custom style. In order to maintain the content of the original video candidate frame image, similar to learning text embeddings that reflect the style * , get the text embedding feature v of the candidate video frame * , and then embedding v according to the text representing the candidate cover and the target style respectively * and * , we can construct a stylized video frame text semantic hint: 'with style s * Image v * ', and input it into the trained image generation model, so that the model can successfully understand the content in the text semantic prompts based on the learning results, and then determine the processing method corresponding to the current candidate cover, and then process the candidate cover to generate a target video cover with a user-specified style.

[0112] Example 3

[0113] A video cover generation device, the device comprising: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the following method steps in embodiment 1:

[0114] Filter the frame images in the target video by using preset filtering conditions to obtain candidate frame images, calculate the video frame features of each candidate frame image; calculate the text features of the text description corresponding to the video frame features;

[0115] A cross-modal interaction method is introduced to measure the semantic relevance between each candidate frame image and the text features; the probability of each candidate frame image being selected as the cover frame is calculated based on the relevance, so as to determine the high-quality video frame that is most relevant to the text description as the candidate cover based on the probability value;

[0116] In response to the cover style specified by the user, the candidate covers are personalized to generate a target cover.

[0117] The frame images in the target video are filtered by the preset filtering conditions as follows:

[0118] Filtering the frame image in the target video with respect to image brightness, clarity and color uniformity to obtain a first filtering result;

[0119] Based on the edge quality change rate of two adjacent frame images that meet the time sequence in the first filtering result, a transition frame image is determined, and the transition frame image is filtered out to obtain a candidate frame image.

[0120] The probability of each candidate frame image becoming a cover frame is determined based on the correlation degree, so as to determine a high-quality video frame that is most relevant to the text description as a candidate cover based on the probability value. Specifically, the following is done:

[0121] Mapping text features and video frame features into the same semantic space; calculating the distance between the text features and the video frame features in the semantic space, and determining the similarity between the text features and the video frame features based on the distance;

[0122] The probability of each candidate frame image being selected as a candidate cover is determined based on similarity calculation; and the candidate frame image with the largest probability value is determined as the candidate cover.

[0123] The device further includes: constructing a target model for selecting a candidate cover, so as to select a candidate cover for the video based on the target model, wherein the loss function of the target model is:

[0124]

[0125] Among them, ||·|| F represents the sum of squares of pixel values, The candidate cover selected by the model, I truth The actual candidate cover.

[0126] In response to the cover style specified by the user, the candidate cover is personalized to generate the target cover:

[0127] Determine a specific word of the cover style specified by the user; map the specific word into a first feature vector based on a word vector model;

[0128] Mapping the text embedding features corresponding to the candidate frame image into a second feature vector based on a word vector model; processing the first feature vector and the second feature vector based on a text encoder to generate a text guide feature;

[0129] Based on the text guidance features and the text embedding features corresponding to the candidate covers, stylized text embedding information matching the cover style specified by the user is generated; based on the stylized text embedding information, the candidate covers are processed to obtain a target cover with the cover style specified by the user.

[0130] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0131] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0132] The data signal is transmitted between the memory 2 and the processor 1 via the bus 3, which will not be described in detail in the embodiment of the present invention.

[0133] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0134] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0135] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0136] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0137] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0138] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0139] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0140] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video cover generation method, characterized in that: The method comprises: Filter the frame images in the target video by using preset filtering conditions to obtain candidate frame images, calculate the video frame features of each candidate frame image; calculate the text features of the text description corresponding to the video frame features; A cross-modal interaction method is introduced to measure the semantic relevance between each candidate frame image and the text features; the probability of each candidate frame image being selected as the cover frame is calculated based on the relevance, so as to determine the high-quality video frame that is most relevant to the text description as the candidate cover based on the probability value; In response to the cover style specified by the user, the candidate covers are personalized to generate a target cover.

2. A video cover generation method according to claim 1, characterized in that: The specific method of filtering the frame images in the target video by using the preset filtering conditions is as follows: Filtering the frame image in the target video with respect to image brightness, clarity and color uniformity to obtain a first filtering result; Based on the edge quality change rate of two adjacent frame images that meet the time sequence in the first filtering result, a transition frame image is determined, and the transition frame image is filtered out to obtain a candidate frame image.

3. A video cover generation method according to claim 1, characterized in that: The method of calculating and determining the selection probability of each candidate frame image becoming a cover frame according to the relevance degree, and determining a high-quality video frame that is most relevant to the text description as a candidate cover based on the probability value, is specifically as follows: Mapping text features and video frame features into the same semantic space; calculating the distance between the text features and the video frame features in the semantic space, and determining the similarity between the text features and the video frame features based on the distance; The probability of each candidate frame image being selected as a candidate cover is determined based on similarity calculation; and the candidate frame image with the largest probability value is determined as the candidate cover.

4. A video cover generation method according to claim 1, characterized in that: The method further includes: constructing a target model for selecting a candidate cover, so as to select a candidate cover for the video based on the target model, wherein the loss function of the target model is: Among them, ||·|| F Represents the square sum of pixel values, I j* The candidate cover selected by the model, I truth The actual candidate cover.

5. A video cover generation method according to claim 1, characterized in that: In response to the cover style specified by the user, the candidate cover is personalized to generate the target cover: Determine a specific word of the cover style specified by the user; map the specific word into a first feature vector based on a word vector model; Mapping the text embedding features corresponding to the candidate frame image into a second feature vector based on a word vector model; processing the first feature vector and the second feature vector based on a text encoder to generate a text guide feature; Based on the text guidance features and the text embedding features corresponding to the candidate covers, stylized text embedding information matching the cover style specified by the user is generated; based on the stylized text embedding information, the candidate covers are processed to obtain a target cover with the cover style specified by the user.

6. A video cover generation device, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.

Images