A method for generating a marketing video with a person as the main character

Abstract

The application discloses a kind of marketing video generation methods with character as main, including the following steps: (1) constructing special character video dataset;(2) constructing special character model network.The present application can better capture the posture features of character movement, and has better performance in character consistency, character shape and posture and limb integrity, solves the problem that the generation effect of the prior art on characters is not ideal, and the problem of often appearing collapse and smearing.

Description

Technical Field

[0001] This invention relates to video processing technology, and more specifically, to a video generation method. Background Technology

[0002] With the rapid development of the internet and e-commerce, more and more merchants are selling goods online. However, marketing videos on e-commerce platforms often require hiring professional photographers to shoot and edit professional models, which is a costly expense for small businesses or individual shops in terms of time and money. At the same time, current video generation models on the market do not produce good results in generating people, often resulting in distorted visuals.

[0003] Currently available video generation models can effectively generate scenic videos, but when people are featured, whether solo or group, unpredictable anomalies such as missing limbs, facial distortions, and movements that defy the laws of physics often occur, resulting in unsatisfactory effects. This poses a significant problem for marketing videos that focus on people, preventing them from fully benefiting from AI. Therefore, developing an algorithm model capable of automatically generating high-quality marketing videos featuring people is imperative. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a marketing video generation method based on people, so as to solve the problem of poor quality of person generation.

[0005] To achieve the above objectives, the technical solution of the present invention is: a method for generating marketing videos based on people, comprising the following steps:

[0006] (1) Construct a dedicated dataset of videos featuring people, including: (101) collecting real marketing videos featuring people, filtering out videos featuring people from a large amount of video data, and eliminating videos featuring non-people; (102) further selecting videos with marketing quality.

[0007] (2) Construct a dedicated character model network, including: (201) Multi-dimensional extraction of character action sequences: deconstruct the input video into a character 3D posture sequence through a human 3D model; deconstruct the input video into a character depth sequence through a depth map model; deconstruct the input video into a character skeleton sequence through an open pose model; deconstruct the input video into a character semantic map sequence through a semantic map model; (202) For the same sequence number frames of the above four sequences, fuse them together through average pooling, and then pass through a nonlinear network for feature cross-interaction, so that the four sequences can be correlated with each other at the same time point to obtain a fused sequence X; (203) For the fused sequence X, use a self-attention model network for feature interaction to obtain the network output Y, and insert the network output Y layer by layer into the semantic segmentation network of the base model for model training, adding an additional character model network to the base model.

[0008] Preferably, step (102) includes: selecting a video set A from four dimensions—color richness (higher than a set threshold), resolution (greater than or equal to 480p), duration (greater than 10 seconds), and screen ratio of the person—that has a color richness higher than a set threshold, a resolution greater than or equal to 480p, a duration greater than 10 seconds, and a screen ratio of the person greater than 30%; training a video embedding model B using a small dataset of high-quality marketing videos collected manually in advance; selecting a small video set as a standard video set that meets the requirements of high-quality marketing videos; converting the standard video into a standard video vector using the video embedding model B; converting video set A into a video set vector; calculating the similarity between the standard video vector and the video set vector one by one; and retaining those with a similarity greater than 0.7.

[0009] Preferably, the method further includes step (3): training the original base model part with a small learning rate, and other training parameters are the same as those of the base model; training the special character model network with twice the learning rate of the base model, and setting twice the loss weight of the loss function for the special character model network.

[0010] Preferably, in step (101), the method for filtering videos containing people from a large amount of video data is as follows: extract the audio track from the video, perform feature analysis on the audio, and extract common sound features; if the sound feature is human voice and is dominant, then the video is judged to be a video that may contain people; the videos marked as "may contain people" are manually verified or reviewed using a more complex model to improve the accuracy.

[0011] Preferably, a preset background sound type that implies the presence of people is set. If the video does not have clear human voice characteristics, and the background sound of the video matches the preset background sound type, then the video is determined to be a video that may contain people.

[0012] Preferably, the sound features include: spectral features, volume variation, pitch, and timbre.

[0013] Existing video generation models are trained for general scenarios. Due to the small proportion of human subjects in the dataset, the influence of human data on the model weights is relatively small, leading to bias from a large amount of other types of data. Furthermore, general networks have weak learning capabilities for human posture and movement. As a result, existing video generation models produce unsatisfactory results for human subjects, often exhibiting issues such as blurriness and misalignment. This invention, however, can better capture the posture features of human movement, demonstrating superior performance in terms of human consistency, body shape, and limb integrity. Attached Figure Description

[0014] The structure of the present invention and its beneficial technical effects will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] Figure 1 This is a flowchart of the method of the present invention;

[0016] Figure 2 This is a flowchart of the process for constructing a dedicated video dataset of people according to the present invention;

[0017] Figure 3 This is a flowchart of the network for constructing a dedicated character model according to the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described in this specification are merely for explaining the invention and are not intended to limit the invention.

[0019] Please see Figure 1 , Figure 2 and Figure 3 The marketing video generation method based on people in this embodiment includes the following steps:

[0020] 1. Steps for constructing a dedicated video dataset of individuals:

[0021] We collect real marketing videos that include people, and use a person recognition model to filter out videos containing people from a large amount of video data, while eliminating videos that do not contain people.

[0022] Further selection of high-quality marketing videos is conducted. The first step involves selecting video set A based on four dimensions: color richness (above a set threshold), resolution (greater than or equal to 480p), duration (greater than 10 seconds), and screen-to-body ratio of the subject (greater than 30%). The second step involves training a video embedding model B (using an embedding model) using a small dataset of approximately 100 high-quality marketing videos (20 seconds each) collected manually beforehand. The third step involves selecting a standard video set that meets the requirements of high-quality marketing videos. The standard videos are converted into embedding vectors using video embedding model B, and video set A is also converted into embedding vectors. The similarity between the standard video embedding vectors and the video set embedding vectors is calculated, and videos with a similarity greater than 0.7 are retained.

[0023] 2. Steps for constructing a dedicated character model network:

[0024] (1) Multi-dimensional extraction of human action sequence: The input video is deconstructed into a human 3D shape map sequence using a human 3D model (using the Skinned Multi-Person Linear model); the input video is deconstructed into a human depth map sequence using a depth map model; the input video is deconstructed into a human skeleton map sequence using an OpenPose model; and the input video is deconstructed into a human semantic map sequence using a semantic map model.

[0025] (2) For the same sequence number frames of the above 4 sequences, they are fused together by mean pooling, and then passed through a ReLU nonlinear network for feature crossing, so that the 4 sequences can be correlated with each other at the same time point to obtain a fused sequence X; for the fused sequence X, a self-attention model network is used for feature interaction, so that the model can learn the key information of the character's action sequence and obtain the network output Y; Y is inserted into the UET network of the base model layer by layer (for different dimensions in each layer of UET, Y is multiplied by a linear network K to obtain the same dimension as UET) for model training, adding an additional character model network to the base model.

[0026] (3) To better learn the feature characterization of the above network for people videos, a classification training method was adopted: the original base model (SVD) was trained with a small learning rate, and other training parameters were the same as the base model; the dedicated people model network was trained with twice the learning rate of the base model, and the loss function was set to twice the loss weight for the dedicated people model network. After multiple rounds of model training, a high-quality people marketing model was finally trained.

[0027] In other implementations, methods for filtering videos containing people from large amounts of video data, besides using person recognition models, can also be performed using scene sound inference methods. This method assumes a correlation between sound features and the presence of people, and indirectly filters videos containing people by inferring these sound features. The method is as follows:

[0028] 1. Sound Feature Extraction

[0029] Audio separation: First, extract the audio track from the video. Tools such as FFmpeg can be used to separate the audio files.

[0030] Audio feature analysis: Perform feature analysis on audio to extract common sound features, such as: spectral features (e.g., MFCC, Mel frequency cepstral coefficients), volume variation (dB), pitch and timbre, etc.

[0031] Human voice and ambient sound classification: Use a simple classifier or rules to detect the presence of human voices in the audio. If human voices are dominant, it can be inferred that there may be people in the video.

[0032] 2. Sound as a basis for inferring the existence of a person.

[0033] Background sound analysis: If there are no clear human voice characteristics, the types of background sounds can be analyzed. Assume certain background sounds might indicate the presence of a person, such as: light footsteps, the sound of a door opening or closing, or the rubbing or colliding of objects.

[0034] Inferring Character Activities: Based on the above sound characteristics, try to infer what activities might have occurred in the scene, thereby indirectly inferring whether any characters were present.

[0035] Videos marked as "may contain people" should be manually verified or reviewed using more sophisticated models to improve accuracy.

[0036] Scene sound inference method infers the presence of people by analyzing the audio features in a video. This method skips the conventional step of directly analyzing the video footage and infers the activities in the scene by changing the sound and its features, thereby indirectly filtering out videos that may contain people. In some special cases, it may have unexpected results.

[0037] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make appropriate changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A method for generating marketing videos focusing on individuals, comprising the following steps: (1) Construct a dedicated video dataset of individuals, including: (101) Collect real marketing advertising videos that include people, filter out videos that include people from a large amount of video data, and eliminate videos that do not include people; (102) Further select videos with marketing quality. (2) Construct a dedicated character model network, including: (201) Multi-dimensional extraction of character action sequences: deconstruct the input video into a character 3D posture sequence through a human 3D model; deconstruct the input video into a character depth sequence through a depth map model; deconstruct the input video into a character skeleton sequence through an OpenPose model; deconstruct the input video into a character semantic map sequence through a semantic map model; (202) For the same sequence number frames of the above four sequences, fuse them together through average pooling, and then pass through a nonlinear network for feature cross-interaction, so that the four sequences can be correlated with each other at the same time point to obtain a fused sequence X; (203) For the fused sequence X, use a self-attention model network for feature interaction to obtain the network output Y. Y is inserted into the Unet network of the base model layer by layer for model training, adding an additional character model network to the base model.

2. The method for generating marketing videos based on individuals according to claim 1, characterized in that, Step (102) includes: selecting a video set A from four dimensions: color richness (higher than a set threshold), resolution (greater than or equal to 480p), duration (greater than 10 seconds), and screen ratio of the person (greater than 30%); training a video embedding model B using a small dataset of high-quality marketing videos of people collected manually in advance; selecting a small video as a standard video set, which meets the requirements of high-quality marketing videos of people; converting the standard video into a standard video vector through the video embedding model B; converting video set A into a video set vector; calculating the similarity between the standard video vector and the video set vector one by one; retaining those with a similarity higher than 0.

7.

3. The method for generating marketing videos based on individuals according to claim 1, characterized in that, It also includes step (3): For the original base model part, a small learning rate is used for training, and other training parameters are the same as those of the base model; for the special character model network, a learning rate of 2 times that of the base is used for training, and the loss function is set to 2 times the loss weight for the special character model network.

4. The method for generating marketing videos based on individuals according to claim 1, characterized in that, In step (101), the method for filtering videos containing people from a large amount of video data is as follows: extract the audio track from the video, perform feature analysis on the audio, and extract sound features; if the sound features are human voices and are dominant, then the video is judged to be a video that may contain people; manually verify the videos marked as "may contain people" or use a more complex model for verification to improve the accuracy.

5. The method for generating marketing videos based on individuals according to claim 4, characterized in that, If a video does not have clear human voice characteristics and the background sound of the video matches the preset background sound type, then the video is determined to be a video that may contain people.

6. The method for generating marketing videos based on individuals according to claim 4, characterized in that, The sound characteristics include: spectral characteristics, volume variation, pitch, and timbre.

Images