Character-based marketing video generation method

Abstract

The invention discloses a character-based marketing video generation method. The method comprises the following steps: (1) constructing a special character video data set; (2) constructing a special character model network; according to the method, the posture characteristics of the character motion can be better captured, the character consistency, the character figure and posture and the limb integrity are better expressed, and the problems that in the prior art, the character generation effect is not ideal, and collapsing and pasting often occur are solved.

Description

Technical Field

[0001] The present invention relates to video processing technology, and more particularly to a video generating method. Background Art

[0002] With the rapid development of the Internet and e-commerce, more and more businesses are selling goods online. However, the marketing videos of merchants on e-commerce platforms often need to hire professional photographers to shoot and edit professional models, which is an expensive cost in terms of time and money for small businesses or individual stores. At the same time, the current video generation models on the market do not generate characters well, and the picture often collapses.

[0003] The video generation models currently available on the market can generate landscape videos relatively well. However, for people in the video, whether it is a single person or multiple people, the results are not satisfactory due to unpredictable abnormal situations such as missing limbs, distorted faces, and movements that do not conform to physical laws. This is a big problem for marketing videos that focus on people, which makes it impossible to enjoy the benefits brought by AI. Therefore, it is imperative to develop an algorithm model that can automatically generate high-quality marketing videos with people. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for generating marketing videos based on characters, so as to solve the problem of poor quality of character generation.

[0005] In order to achieve the above object, the technical solution of the present invention is: a method for generating a marketing video based on characters, comprising the following steps:

[0006] (1) Constructing a dedicated person video dataset, including: (101) collecting real marketing advertising videos containing people, screening out videos containing people from a large amount of video data, and eliminating non-person videos; (102) further selecting videos with marketing quality;

[0007] (2) Constructing a dedicated character model network, including: (201) extracting character action sequences in multiple dimensions: deconstructing the input video through a three-dimensional human body model to obtain a character 3D body graph sequence; deconstructing the input video through a depth graph model to obtain a character depth graph sequence; deconstructing the input video through an open posture model to obtain a character skeleton graph sequence; deconstructing the input video through a semantic graph model to obtain a character semantic graph sequence; (202) fusing the same sequence number frames of the above four sequences together by average pooling, and then performing feature crossover through a layer of nonlinear network, so that the four sequences can be correlated with each other at the same time point, and obtaining a fused sequence X; (203) for the fused sequence X, using a self-attention model network for feature interaction to obtain a network output Y, and inserting the network output Y layer by layer into the semantic segmentation network of the base model for model training, thereby adding an additional character model network to the base model.

[0008] Preferably, in step (102), it includes: selecting a video set A with a color richness higher than a set threshold, a resolution greater than or equal to 480p, a duration greater than 10 seconds, and a character screen ratio greater than 30% from four dimensions of color richness, resolution, duration, and character screen ratio; training a video embedding model B using a small data set of high-quality character marketing videos collected manually in advance; selecting a small video as a standard video set that meets the requirements of high-quality character marketing videos, converting the standard video into a standard video vector through the video embedding model B, converting the video set A into a video set vector, calculating the similarity between the standard video vector and the video set vector one by one, and retaining those with a similarity greater than 0.7.

[0009] Preferably, the method further includes step (3): training the original base model part with a small learning rate, with other training parameters being the same as those of the base model; training the dedicated character model network with a learning rate twice that of the base, and setting a loss weight of twice that of the dedicated character model network in the loss function.

[0010] Preferably, in step (101), the method for selecting videos containing people from a large amount of video data is as follows: extracting an audio track from the video, performing feature analysis on the audio, and extracting common sound features; if the sound features are human voices and are dominant, then the video is judged to be a video that may contain people; and manually verifying the videos marked as "may contain people" or reviewing them using a more complex model to improve the accuracy.

[0011] Preferably, a preset background sound type is set that implies the presence of a person. If the video does not have a clear human voice feature, if the background sound of the video matches the preset background sound type, the video is determined to be a video that may contain a person.

[0012] Preferably, the sound characteristics include: spectrum characteristics, volume change, pitch and timbre.

[0013] Existing video generation models are all trained for general scenarios. Due to the small proportion of character data sets, the model weight affected by character data is small, and there is a possibility of being biased by a large amount of other types of data. In addition, the general network has a weak learning ability for character posture and movement. The existing video generation model has an unsatisfactory effect on character generation, and often crashes and becomes blurry. The present invention can better capture the posture characteristics of character movement, and has better performance in character consistency, character body shape and limb integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The structure of the present invention and its beneficial technical effects are described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0015] Figure 1 is a flow chart of the method of the present invention;

[0016] Figure 2 It is a flow chart of constructing a dedicated character video data set of the present invention;

[0017] Figure 3 It is a network flow chart of building a special character model of the present invention. DETAILED DESCRIPTION

[0018] In order to make the invention purpose, technical solution and beneficial technical effects of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the specific implementation methods described in this specification are only for explaining the present invention, not for limiting the present invention.

[0019] See also Figure 1 , Figure 2 and Figure 3 The method for generating a marketing video based on characters in this embodiment includes the following steps:

[0020] 1. Steps for constructing a dedicated person video dataset:

[0021] Collect real marketing advertising videos containing people, filter out videos containing people from a large amount of video data through a person recognition model, and eliminate non-person videos.

[0022] Further select videos with marketing quality. The first step is to select video set A from four dimensions: color richness, resolution, duration, and character screen ratio. The video set A has color richness higher than the set threshold, resolution greater than or equal to 480p, duration greater than 10 seconds, and character screen ratio greater than 30%; the second step is to use a small data set of high-quality character marketing videos collected manually in advance, about 100 20s videos, to train a video embedding model B (select embedding model); the third step is to select a small video as the standard video set, which meets the requirements of high-quality character marketing videos, and convert the standard video into embedding vectors through video embedding model B. The video set A is also converted into embedding vectors. The standard video embedding vectors are calculated one by one for similarity with the video set embedding vectors, and those with similarity higher than 0.7 are retained.

[0023] 2. Steps to build a dedicated character model network:

[0024] (1) Multi-dimensional extraction of character action sequences: The input video is deconstructed into a character 3D shape map sequence through a human 3D model (using the Skinned Multi-Person Linear model); the input video is deconstructed into a character depth map sequence through a depth map model; the input video is deconstructed into a character skeleton map sequence through an openpose model; the input video is deconstructed into a character semantic map sequence through a semantic map model.

[0025] (2) The frames with the same sequence number of the above four sequences are fused together by mean pooling, and then passed through a layer of relu nonlinear network for feature crossover, so that the four sequences can be correlated with each other at the same time point to obtain a fused sequence X; for the fused sequence X, the self-attention model network is used for feature interaction, so that the model can learn the key information of the character's action sequence and obtain the network output Y; Y is inserted layer by layer (concat) into the unet network of the base model (for different dimensions in each layer of unet, Y is multiplied by a linear network K to obtain the same dimension as unet), and model training is performed to add an additional character model network to the base model.

[0026] (3) In order to better learn the characteristics of the above network in characterizing the video of the person, a classification training method is adopted: for the original base model (SVD), a small learning rate is used for training, and other training parameters are the same as the base model; for the dedicated character model network, a learning rate twice that of the base is used for training, and the loss function sets a loss weight of twice for the dedicated character model network. After multiple rounds of model training, a high-quality character marketing model is finally trained.

[0027] In other implementations, in addition to using a person recognition model, the method for selecting videos containing people from a large amount of video data can also be performed by using a scene sound inversion method. This method assumes that there is some correlation between sound features and the existence of people, and indirectly selects videos containing people by inverting these sound features. The method is as follows:

[0028] 1. Sound feature extraction

[0029] Audio separation: First, extract the audio track from the video. You can use tools such as FFmpeg to separate the audio file.

[0030] Audio feature analysis: Perform feature analysis on audio to extract common sound features, such as spectral features (such as MFCC, Mel-frequency cepstral coefficients), volume change (dB), pitch and timbre, etc.

[0031] Human and environmental sound classification: Use simple classifiers or rules to detect whether there is human voice in the audio. If the human voice is dominant, it can be inferred that there may be people in the video.

[0032] 2. Sound can infer the existence of characters

[0033] Background sound analysis: If there is no clear human voice feature, the type of background sound can be analyzed. Assume that some background sounds may indicate the presence of a person, such as light footsteps, the sound of a door opening, the sound of objects rubbing or colliding, etc.

[0034] Inferring character activities: Based on the above sound features, try to infer what activities may have occurred in the scene, and thus indirectly infer whether there are people present.

[0035] Videos marked as "may contain people" are manually verified or reviewed using more complex models to improve accuracy.

[0036] The scene sound inference method infers the presence of people by analyzing the audio features in the video. This method skips the conventional step of directly analyzing the video screen and infers the activities in the scene through the changes and features of the sound, thereby indirectly screening out videos that may contain people. In some special cases, this may have unexpected effects.

[0037] According to the disclosure and teaching of the above description, those skilled in the art to which the present invention belongs may also make appropriate changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the scope of protection of the claims of the present invention. In addition, although some specific terms are used in this specification, these terms are only for the convenience of description and do not constitute any limitation to the present invention.

Claims

1. A method for generating a marketing video based on characters, comprising the following steps: (1) Construct a dedicated person video dataset, including: (101) Collect real marketing advertising videos containing people, filter out videos containing people from a large amount of video data, and exclude videos without people; (102) Further select videos with marketing quality; (2) Constructing a dedicated character model network, including: (201) extracting character action sequences in multiple dimensions: deconstructing the input video through a three-dimensional human body model to obtain a character 3D body graph sequence; deconstructing the input video through a depth graph model to obtain a character depth graph sequence; deconstructing the input video through an open posture model to obtain a character skeleton graph sequence; deconstructing the input video through a semantic graph model to obtain a character semantic graph sequence; (202) fusing the same sequence number frames of the above four sequences together by average pooling, and then performing feature crossover through a layer of nonlinear network, so that the four sequences can be correlated with each other at the same time point, and obtaining a fused sequence X; (203) for the fused sequence X, using a self-attention model network for feature interaction to obtain a network output Y, and inserting the network output Y layer by layer into the semantic segmentation network of the base model for model training, thereby adding an additional character model network to the base model.

2. The method for generating a marketing video based on characters according to claim 1, characterized in that: In step (102), it includes: selecting a video set A with a color richness higher than a set threshold, a resolution greater than or equal to 480p, a duration greater than 10 seconds, and a character screen ratio greater than 30% from four dimensions of color richness, resolution, duration, and character screen ratio; training a video embedding model B using a small data set of high-quality character marketing videos collected manually in advance; selecting a small video as a standard video set that meets the requirements of high-quality character marketing videos, converting the standard video into a standard video vector through the video embedding model B, converting the video set A into a video set vector, calculating the similarity between the standard video vector and the video set vector one by one, and retaining those with a similarity greater than 0.

7.

3. The method for generating a marketing video based on characters according to claim 1, characterized in that: The method also includes step (3): for the original base model part, a small learning rate is used for training, and other training parameters are the same as the base model; for the dedicated character model network, a learning rate twice that of the base is used for training, and the loss function sets a loss weight of twice that of the dedicated character model network.

4. The method for generating a marketing video based on characters according to claim 1, characterized in that: In step (101), the method for selecting videos containing people from a large amount of video data is as follows: extracting an audio track from the video, performing feature analysis on the audio, and extracting common sound features; if the sound features are human voices and are dominant, then the video is judged to be a video that may contain people; and manually verifying the videos marked as "may contain people" or using a more complex model for review to improve the accuracy.

5. The method for generating a marketing video based on characters according to claim 4, characterized in that: A preset background sound type is set that indicates the presence of a person. If the video does not have clear human voice features, if the background sound of the video matches the preset background sound type, the video is determined to be a video that may contain a person.

6. The method for generating a marketing video based on characters according to claim 4, characterized in that: The sound features include: frequency spectrum features, volume changes, pitch and timbre.

Images