Method and device for generating high-quality figure video for fixing portrait and fitting oral broadcast scene

Abstract

The invention provides a high-quality figure video generation method and device for fixing a portrait and fitting an oral broadcast scene, and the method comprises the steps: obtaining a personal photo provided by a user, calling a preset real person portrait generation model to carry out the preprocessing of the personal photo, and generating diversified portraits; constructing a prompt template library, calling a video generation model to select a corresponding prompt template from the prompt template library according to the diversified portraits, generating portrait videos, and performing automatic screening processing on the generated portrait videos to obtain a material library; and collecting user feedback, and optimizing the prompt template library according to the user feedback. The invention aims to solve the problems of unreliable effect, false details, high difficulty, high failure rate, fixed action, fixed makeup and fixed clothes in the existing portrait video generation scheme.

Description

Technical Field

[0001] The present invention relates to the technical field of portrait video generation, and in particular to a method and device for generating high-quality character videos with fixed portraits and suitable for oral broadcasting scenes. Background Art

[0002] The current solutions for generating task images with large models commonly used in the market are not realistic enough, and their sense of appeal and attraction do not always meet the requirements of use; specifically, the generated character images may not be fine enough in detail processing, and it is difficult to meet the high-value standards generally recognized by the public, and the clothing and makeup effects may also lack diversity and realism. At the same time, the angles of video / photo shooting and character movements are single. Most of the generated videos use fixed shooting angles, lack changes, and it is difficult to achieve real lens movement effects, such as push-pull, shaking, etc.; and the character makeup and clothing are single, and the emotional expression, action, background, and lighting effects are single. The realism and dynamic changes of the generated background are usually fake and the degree of integration is not high due to the background obtained by the method of cutting out the image to change the background. There are also copyright and privacy issues. This is because the frequent use of public videos on the Internet may involve risks of portrait rights and privacy rights; the works generated by changing the expressions and actions of characters in the video may have originality problems due to excessive similarity. Finally, this solution is difficult to implement technically and has high costs when used in practice.

[0003] Specifically, existing portrait video generation solutions mostly use the portrait + prompt method to generate oral short video materials, which has problems such as unreliable effects, false details, high difficulty, and high failure rate. At the same time, the limb-driven solution adopted by this solution has problems of fixed movements, fixed makeup, and fixed clothing.

[0004] In view of this, this application is filed. Summary of the invention

[0005] The present invention provides a method and device for generating high-quality character videos with fixed portraits and suitable for oral broadcast scenes, which can at least partially improve the above-mentioned problems.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for generating high-quality character videos with fixed portraits and suitable for oral broadcasting scenes, comprising:

[0008] Obtaining personal photos provided by the user, calling a preset real-person portrait generation model to pre-process the personal photos to generate diversified portraits;

[0009] Constructing a prompt template library, calling a video generation model to select a corresponding prompt template from the prompt template library according to the diverse portraits, generating portrait videos, and automatically screening the generated portrait videos to obtain a material library;

[0010] Collect user feedback, and optimize the prompt template library according to the user feedback.

[0011] The present invention also provides a high-quality character video generation device that fixes the human portrait and fits the oral broadcast scene, which includes:

[0012] A portrait material generation unit, used to obtain personal photos provided by a user, call a preset real-person portrait generation model to pre-process the personal photos, and generate diversified portraits;

[0013] A portrait video generation unit is used to build a prompt template library, call a video generation model to select a corresponding prompt template from the prompt template library according to the diverse portraits, generate a portrait video, and automatically screen the generated portrait video to obtain a material library;

[0014] A feedback unit is used to collect user feedback and optimize the prompt template library according to the user feedback.

[0015] In summary, the method for generating high-quality character videos with fixed portraits and suitable for oral broadcast scenarios can be combined with business to batch generate videos that meet business requirements; and this method combines the innovative capabilities of large models to improve the diversity of character videos and avoid the risk of repeated use of video materials being judged as highly repetitive materials by some platforms; at the same time, it also reduces the time and economic costs of manual shooting and production throughout the entire process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a method for generating a high-quality character video with a fixed portrait and adapted to a spoken scene provided by the first embodiment of the present invention;

[0017] Figure 2 It is a module schematic diagram of a device for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] refer to Figure 1As shown, the first embodiment of the present invention discloses a method for generating a high-quality character video with a fixed portrait and a scene that fits the oral broadcast, which can be executed by a high-quality character video generating device with a fixed portrait and a scene that fits the oral broadcast (hereinafter referred to as a generating device), and in particular, executed by one or more processors in the generating device to implement the following method:

[0020] S101, obtaining a personal photo provided by a user, calling a preset real-person portrait generation model to pre-process the personal photo, and generating diversified portraits;

[0021] Specifically, step S101 includes: obtaining prompt design experience, wherein the prompt design experience is obtained by summarizing the initial prompt after multiple experimental adjustments;

[0022] Using the LLM model to process according to the prompt design experience, multiple prompts that meet the requirements of the preset spoken character images are generated;

[0023] Obtaining variable information, wherein the variable information includes different screen ratio requirements, different backgrounds, different clothing types and colors, different makeup, and different character postures;

[0024] On the basis of multiple prompts that meet the requirements of the preset spoken character image, one or more combinations of variables in the variable information are added to ensure that the prompts in each combination are different;

[0025] A multimodal model is used to screen the prompts after adding variables, select the prompts with the highest success rate, and generate diverse portraits.

[0026] The multimodal model is used to filter the prompts after adding variables to select the prompts with the highest success rate, specifically:

[0027] Obtaining high-quality portrait requirements, calling a multimodal model to make a judgment based on the high-quality portrait requirements, and judging whether the current portrait image meets the high-quality portrait requirements;

[0028] Record the prompt corresponding to the portrait image that meets the high-quality portrait requirements determined by the multimodal model;

[0029] Count the success rate of each matching prompt and keep the prompt with the highest success rate.

[0030] The multimodal model is called to make a judgment according to the high-quality portrait requirement to determine whether the current portrait image meets the high-quality portrait requirement, specifically:

[0031] The characteristics of the characters in the high-quality oral broadcast video are used as a judgment basis, and the multimodal model makes a judgment based on the judgment basis;

[0032] A small model is used to correct the front face, clarity, and the proportion of the person in the screen. Among them, a binary classification model is used to judge the front face, a binary classification model is used to judge the clarity, and a portrait segmentation model is used to calculate the proportion of the person's pixels.

[0033] In this embodiment, portrait materials are generated; first, personal photos provided by the user are obtained, and then the portrait generation model is called to generate more portraits. The portrait generation model can be a self-developed model, an open source model, or a paid API, and supports a high-quality real-person portrait generation model of image + prompt raw image.

[0034] Specifically, in this embodiment, the step of generating diversified portraits includes:

[0035] The first step is prompt design: manually design prompts, and then conduct multiple experiments to optimize and summarize experience; based on experience, use LLM to generate a batch of prompts that meet the requirements of spoken character images. Among them, the requirements for the characteristics of characters in high-quality spoken video, taking fashion and clothing short videos as an example, include: 1. Full face, high definition (1080P or above), 2. The proportion of the character to the screen (full-length image: about 1 / 5; half-length image: 1 / 5~1 / 3; close-up: 1 / 3~1 / 2), 3. Fresh and natural makeup, 4. Fashionable and generous clothes, avoid being too fancy or monotonous, 5. Look more friendly, natural, smiling and confident, 6. Appropriate posture and expression, 7. Background: different lighting, indoor and outdoor, background decoration (soft lighting, relatively clean background).

[0036] In this embodiment, a prompt example is given: a fashionable female image is generated, with a confident smile, fresh and natural makeup, fashionable casual clothes, coordinated colors, and simple earrings and watches. The background is a modern indoor scene, including simple furniture and green plants, soft lighting, and a slightly blurred background to highlight the character.

[0037] The second step is to generate diverse portraits: obtain variables, which can be: different screen ratio requirements, different backgrounds, different clothing types and colors, different makeup, and character postures. After the prompt is determined, add a combination of one or more of the above variables to ensure that each prompt is different and generate diverse portraits.

[0038] The third step is screening: you can ask for prompts that can generate high-quality images and have a high success rate in manual screening; you can also use a multimodal model to screen, first tell the multimodal large model the requirements for high-quality portraits, and let the large model judge whether the current image meets the requirements. The basis for judgment is: refer to the characteristics of the characters in the above high-quality oral videos (taking fashion and clothing type short videos as an example). Let the large model draw conclusions based on the judgment basis; for dimensions such as front face, clarity, and the proportion of characters in the screen, you can also use small models to do verification: front face judgment-two-classification model; clarity judgment-two-classification model; character proportion: portrait segmentation, calculate the proportion of character pixels (error within 5% can be tolerated).

[0039] Record the prompts corresponding to the images that the big model thinks are reasonable, count the success rate of each prompt, and keep the prompts with high success rates. The success rate of the prompt is determined according to the specific situation (the reference value of the success rate is 70%).

[0040] S102, constructing a prompt template library, calling a video generation model to select a corresponding prompt template from the prompt template library according to the diverse portraits, generating a portrait video, and automatically screening the generated portrait video to obtain a material library;

[0041] Specifically, step S102 includes: browsing various types of oral videos, summarizing the video features of each video category, and obtaining a video description summary, wherein the oral videos are divided and classified according to the industry;

[0042] Call the LLM model to generate prompts with differentiated features in batches according to the video description summary;

[0043] The batch-generated prompts with differentiated characteristics are screened according to the quality of the video generation, and the prompts that meet the preset requirements are added to the prompt word template library to generate a prompt template library.

[0044] Calling the LLM model to generate prompts with differentiated characteristics in batches according to the video description summary, specifically: obtaining existing differentiated oral portrait materials, calling the LLM model to understand the existing differentiated oral portrait materials, and understanding the characteristics of the characters in the portraits, whose dimensions are consistent with the characteristics of the characters in the portraits mentioned in the process of constructing the portrait prompts;

[0045] Randomly select at least one variable from the candidate variables, add the selected variable to the prompt, and generate a prompt with differentiated characteristics, wherein the candidate variables include: sitting / standing, still / walking, front face / side face, emotional atmosphere, and camera movement.

[0046] The video generation model is called to select a corresponding prompt template from the prompt template library according to the diverse portraits to generate a portrait video, specifically:

[0047] Calling a video generation model, selecting a corresponding prompt template from the prompt template library according to the diverse portraits, and generating a video;

[0048] Adjust the prompt according to the video generation effect, optimize the effect, and regenerate the portrait video.

[0049] The generated portrait videos are automatically screened and processed to obtain a material library, specifically:

[0050] Extract frames from the generated portrait video to determine the video clarity, inter-frame continuity, and abnormal character distortion;

[0051] Filter out the portrait videos that meet the requirements and obtain the material library.

[0052] In this embodiment, the generation of portrait videos is continued, wherein the video generation model may adopt models such as Zhipu Qingyan, kling, sora, etc., and then the model is fine-tuned.

[0053] To build a prompt template library, first obtain the video description summary, browse various oral videos, and summarize the video characteristics of each video category. Video categories are divided by industry, such as education and training, emotion, health, clothing, etc.

[0054] LLM generates differentiated prompts in batches based on the video characteristics of each industry category described in the video; specifically, the provided portrait is already a differentiated oral portrait material, allowing the big model to understand the characteristics of the person in the portrait. The dimensions are consistent with the portrait characteristics mentioned in the portrait prompt construction process, and the following variables are randomly selected: sitting / standing, still / walking, frontal / side, emotional atmosphere (warm and harmonious, lively and lively), camera movement (horizontal, vertical, close-up, far-away), etc., including but not limited to the above variables, randomly select at least one variable and add it to the prompt. Subsequently, the video generation quality is manually screened, and the prompts that are considered to meet the requirements are added to the prompt word template library.

[0055] To generate a video, first select the model for video generation, and use the designed prompt template to generate a video; adjust the prompt according to the generated effect, optimize the effect, and regenerate. When screening the video, you can get the material library, which can be screened manually or automatically; the automatic screening includes: extracting frames to determine clarity, inter-frame continuity (calculating the similarity of the color histogram between frames, judging whether it is discontinuous based on the threshold, which is obtained through normal video statistics), and whether the characters are abnormally distorted.

[0056] S103, collecting user feedback, and optimizing the prompt template library according to the user feedback.

[0057] Specifically, step S103 includes: collecting user feedback, and optimizing the portrait generation prompt template library and the video generation prompt template library in a targeted manner according to the user feedback, wherein the variables of the prompts that mainly optimize the characteristics of the character images and improve the diversity of the generated videos are mainly optimized.

[0058] In this embodiment, feedback is finally conducted to collect user feedback and optimize the portrait generation prompt template library and the video generation prompt template library in a targeted manner (mainly optimizing the characteristics of the character image and improving the variables of the prompt for generating video diversity).

[0059] In summary, the method for generating high-quality character videos with fixed portraits and suitable for oral broadcast scenes designs a solution for generating diverse video materials based on character pictures. This method can be combined with business to batch generate videos that meet business requirements. This embodiment takes the generation of oral broadcast video materials as an example to introduce the solution. It combines the innovative capabilities of large models to improve the diversity of character videos, avoids the risk of video materials being reused and judged as highly repeated materials by some platforms, and also reduces the time and economic costs of manual shooting and production throughout the process.

[0060] See also Figure 2 The second embodiment of the present invention further provides a device for generating a high-quality character video with a fixed portrait and adapted to a spoken scene, which includes:

[0061] The portrait material generation unit 201 is used to obtain personal photos provided by the user, call a preset real-person portrait generation model to pre-process the personal photos, and generate diversified portraits;

[0062] The portrait video generation unit 202 is used to build a prompt template library, call the video generation model to select a corresponding prompt template from the prompt template library according to the diverse portraits, generate a portrait video, and automatically screen the generated portrait video to obtain a material library;

[0063] The feedback unit 203 is used to collect user feedback and optimize the prompt template library according to the user feedback.

[0064] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating high-quality character videos with fixed portraits and suitable for oral broadcasting scenes, characterized in that: include: Obtaining personal photos provided by the user, calling a preset real-person portrait generation model to pre-process the personal photos to generate diversified portraits; Constructing a prompt template library, calling a video generation model to select a corresponding prompt template from the prompt template library according to the diverse portraits, generating portrait videos, and automatically screening the generated portrait videos to obtain a material library; Collect user feedback, and optimize the prompt template library according to the user feedback.

2. The method for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes according to claim 1 is characterized in that: The preset real-person portrait generation model is called to pre-process the personal photo to generate diversified portraits, specifically: Obtain prompt design experience, wherein the prompt design experience is obtained by summarizing the initial prompt after multiple experimental adjustments; Using the LLM model to process according to the prompt design experience, multiple prompts that meet the requirements of the preset spoken character images are generated; Obtaining variable information, wherein the variable information includes different screen ratio requirements, different backgrounds, different clothing types and colors, different makeup, and different character postures; On the basis of multiple prompts that meet the requirements of the preset spoken character image, one or more combinations of variables in the variable information are added to ensure that the prompts in each combination are different; A multimodal model is used to screen the prompts after adding variables, select the prompts with the highest success rate, and generate diverse portraits.

3. The method for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes according to claim 2 is characterized in that: The multimodal model is used to filter the prompts after adding variables to select the prompts with the highest success rate, specifically: Obtaining high-quality portrait requirements, calling a multimodal model to make a judgment based on the high-quality portrait requirements, and judging whether the current portrait image meets the high-quality portrait requirements; Record the prompt corresponding to the portrait image that meets the high-quality portrait requirements determined by the multimodal model; Count the success rate of each matching prompt and keep the prompt with the highest success rate.

4. The method for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes according to claim 3 is characterized in that: The multimodal model is called to make a judgment according to the high-quality portrait requirement to determine whether the current portrait image meets the high-quality portrait requirement, specifically: The characteristics of the characters in the high-quality oral broadcast video are used as a judgment basis, and the multimodal model makes a judgment based on the judgment basis; A small model is used to correct the front face, clarity, and the proportion of the person in the screen. Among them, a binary classification model is used to judge the front face, a binary classification model is used to judge the clarity, and a portrait segmentation model is used to calculate the proportion of the person's pixels.

5. The method for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes according to claim 1, characterized in that: Build a prompt template library, specifically: Browse various types of oral videos, summarize the video features of each video category, and obtain a video description summary. Among them, oral videos are divided and classified according to industry; Call the LLM model to generate prompts with differentiated features in batches according to the video description summary; The batch-generated prompts with differentiated characteristics are screened according to the quality of the video generation, and the prompts that meet the preset requirements are added to the prompt word template library to generate a prompt template library.

6. The method for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes according to claim 5, characterized in that: Call the LLM model to generate prompts with differentiated features in batches according to the video description summary, specifically: Obtain existing differentiated oral portrait materials, call the LLM model to understand the existing differentiated oral portrait materials, and understand the characteristics of the characters in the portraits, whose dimensions are consistent with the characteristics of the characters in the portraits mentioned in the portrait prompt construction process; Randomly select at least one variable from the candidate variables, add the selected variable to the prompt, and generate a prompt with differentiated characteristics, wherein the candidate variables include: sitting / standing, still / walking, front face / side face, emotional atmosphere, and camera movement.

7. The method for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes according to claim 1, characterized in that: The video generation model is called to select a corresponding prompt template from the prompt template library according to the diverse portraits to generate a portrait video, specifically: Calling a video generation model, selecting a corresponding prompt template from the prompt template library according to the diverse portraits, and generating a video; Adjust the prompt according to the video generation effect, optimize the effect, and regenerate the portrait video.

8. The method for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes according to claim 1, characterized in that: The generated portrait videos are automatically screened and processed to obtain a material library, specifically: Extract frames from the generated portrait video to determine the video clarity, inter-frame continuity, and abnormal character distortion; Filter out the portrait videos that meet the requirements and obtain the material library.

9. The method for generating high-quality character videos with fixed portraits and adapted to oral broadcast scenes according to claim 1, characterized in that: Collect user feedback and optimize the prompt template library according to the user feedback, specifically: Collect user feedback, and optimize the portrait generation prompt template library and the video generation prompt template library in a targeted manner based on the user feedback, among which the variables of the prompt that mainly optimizes the characteristics of the character image and improves the diversity of the generated video.

10. A high-quality character video generation device with a fixed portrait and a scene that fits the spoken scene, characterized in that: include: A portrait material generation unit, used to obtain personal photos provided by a user, call a preset real-person portrait generation model to pre-process the personal photos, and generate diversified portraits; A portrait video generation unit is used to build a prompt template library, call a video generation model to select a corresponding prompt template from the prompt template library according to the diverse portraits, generate a portrait video, and automatically screen the generated portrait video to obtain a material library; A feedback unit is used to collect user feedback and optimize the prompt template library according to the user feedback.

Images