Diversity optimization and automatic optimization method and device for broadcast video

Abstract

The invention provides a diversity optimization and automatic optimization method and device for an oral video, and the method comprises the steps: obtaining a personal photo provided by a user, calling a pre-trained portrait generation model to carry out the preprocessing of the personal photo, generating diversified portraits, and carrying out the optimization of the diversified portraits; carrying out extraction processing from a preset oral playing video library, inputting the extracted picture with the figure and the copywriting information into a multi-mode large model for summarization, and generating an oral playing video; and when it is judged that the generated oral playing video has an abnormal problem, performing abnormal problem optimization processing on the oral playing video. The invention aims to solve the problems that the workload of manual screening and manual experience summarization is more used in the existing portrait video generation scheme, and the efficiency is low; and the method is more dependent on manual experience and has limitation.

Description

Technical Field

[0001] The present invention relates to the field of portrait video technology, and in particular to a method and device for diversity optimization and automatic optimization of spoken video. Background Art

[0002] The current solutions for generating task images with large models are not realistic enough, and are not always attractive enough to meet the requirements of use. Specifically, the generated characters may not be detailed enough to meet the high appearance standards generally recognized by the public, and the clothing and makeup effects may lack diversity and realism. At the same time, the angles of video / photo shooting and the movements of characters are single.

[0003] Most of the generated videos use fixed shooting angles, lack of changes, and it is difficult to achieve real lens movement effects, such as push-pull, shaking, etc.; and the makeup and clothing of the characters are monotonous, and the emotional expression, movements, backgrounds, and lighting effects are monotonous. The realism and dynamic changes of the generated background are usually fake and the degree of integration is not high due to the background obtained by the method of cutting out the image to change the background. There are also issues of copyright and privacy. This is because there may be risks of portrait rights and privacy rights due to the frequent use of public videos on the Internet; works generated by changing the expressions and movements of the characters in the video may have originality problems due to excessive similarity. Finally, when this solution is actually used, there are still problems with technical implementation and high costs.

[0004] Specifically, existing portrait video generation solutions rely heavily on manual screening and manual experience summarization, which is inefficient; they rely more on manual experience and have limitations.

[0005] In view of this, this application is filed. Summary of the invention

[0006] The present invention provides a method and device for diversity optimization and automatic optimization of spoken video, which can at least partially improve the above-mentioned problems.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for optimizing diversity and automation of oral video, comprising:

[0009] Obtain personal photos provided by users, build a high-quality single-person oral video library, a high-quality oral character image feature library, and a diversified character image library, and optimize the generation of portraits;

[0010] Extract and process the pre-set spoken video library, and input the extracted pictures and text information with characters into the multi-modal large model for summarization to generate spoken video;

[0011] When it is determined that the generated spoken-word video has abnormal problems, the spoken-word video is optimized for the abnormal problems.

[0012] The present invention also provides a device for optimizing the diversity and automation of oral video, which comprises:

[0013] A diversified portrait optimization unit is used to obtain personal photos provided by users, and to construct a high-quality single-person oral video library, a high-quality oral character image feature library, and a diversified character image library, respectively, to optimize the generation of portraits;

[0014] The spoken video optimization unit is used to extract and process the images and text information with characters from the preset spoken video library, and input the extracted images and text information with characters into the multimodal large model for summarization to generate the spoken video;

[0015] The abnormal problem optimization unit is used to optimize the abnormal problem of the spoken video when it is determined that the generated spoken video has an abnormal problem.

[0016] In summary, the diversity optimization and automated optimization method of the voice-over video is combined with the business to batch generate videos that meet business requirements. At the same time, the generated diverse portraits and voice-over videos are optimized, reducing the time and economic costs of manual shooting and production throughout the entire process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a method for diversity optimization and automatic optimization of oral video provided by the first embodiment of the present invention;

[0018] Figure 2 It is a module diagram of a device for diversity optimization and automatic optimization of spoken video provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] refer to Figure 1 As shown, the first embodiment of the present invention discloses a method for diversity optimization and automatic optimization of spoken video, which can be executed by a diversity optimization and automatic optimization device for spoken video (hereinafter referred to as an optimization device), and in particular, executed by one or more processors in the optimization device to implement the following method:

[0021] S101, obtaining personal photos provided by the user, respectively constructing a high-quality single-person oral video library, a high-quality oral character image feature library, and a diversified character image library, to optimize the generation of portraits;

[0022] Specifically, step S101 includes: constructing a high-quality single-person oral video library, wherein the video data under each commodity category is sorted in descending order according to the number of likes, and the top k videos are taken and stored as high-quality oral videos of the category, thereby obtaining a high-quality single-person oral video library;

[0023] Construct a high-quality oral broadcast character image feature library, where multiple videos of each product category are extracted, and one video frame is extracted. The extracted images are summarized using a multimodal large model. One oral broadcast video has one character feature description, which will be used as a character image prompt, expressed as prompt, to obtain a high-quality oral broadcast character image feature library;

[0024] Build a diverse character image library, which uses a multimodal large model to interact with the image generation model multiple times, tune the prompt word, generate character image pictures, pre-process each character, generate images of different styles, store them in the library, and perform screening and filtering to achieve secondary quality assurance.

[0025] Use the multimodal large model to interact with the image generation model multiple times, tune the prompt word prompt, and generate character image pictures, specifically:

[0026] Extract a prompt word prompt, use the preset text-based graph model to process the prompt word prompt, and generate a character image;

[0027] Use a large multimodal model to judge the quality of the image and the difference between the image content and the prompt word;

[0028] When it is determined that the quality of the image does not meet the preset conditions, the prompt word is optimized, a prompt word is re-extracted, the prompt word is processed using the preset text-based image model, and a character image is generated for judgment;

[0029] When it is determined that the quality of the image meets the preset conditions, the image of the person is saved;

[0030] Each prompt can only be repeated 10 times at most. If the person still does not meet the preset conditions, the prompt will be discarded.

[0031] When it is determined that a total of 5 character images are saved, stop and obtain a preliminary character image library, in which each character in the library is a different character.

[0032] Preprocess each character to generate images of different styles, specifically:

[0033] Obtain prompt design experience, wherein the prompt design experience is obtained by summarizing the initial prompt after multiple experimental adjustments;

[0034] Using the LLM model to process according to the prompt design experience, multiple prompts that meet the requirements of the preset spoken character images are generated;

[0035] Obtaining variable information, wherein the variable information includes different screen ratio requirements, different backgrounds, different clothing types and colors, different makeup, and different character postures;

[0036] On the basis of multiple prompts that meet the requirements of the preset spoken character image, one or more combinations of variables in the variable information are added to ensure that the prompts in each combination are different;

[0037] A multimodal model is used to screen the prompts after adding variables, select the prompts with the highest success rate, and generate diverse portraits.

[0038] The multimodal model is used to filter the prompts after adding variables to select the prompts with the highest success rate, specifically:

[0039] Obtaining high-quality portrait requirements, calling a multimodal model to make a judgment based on the high-quality portrait requirements, and judging whether the current portrait image meets the high-quality portrait requirements;

[0040] Record the prompt corresponding to the portrait image that meets the high-quality portrait requirements determined by the multimodal model;

[0041] Count the success rate of each matching prompt and keep the prompt with the highest success rate.

[0042] The multimodal model is called to make a judgment according to the high-quality portrait requirement to determine whether the current portrait image meets the high-quality portrait requirement, specifically:

[0043] The characteristics of the characters in the high-quality oral broadcast video are used as a judgment basis, and the multimodal model makes a judgment based on the judgment basis;

[0044] A small model is used to correct the front face, clarity, and the proportion of the person in the screen. Among them, a binary classification model is used to judge the front face, a binary classification model is used to judge the clarity, and a portrait segmentation model is used to calculate the proportion of the person's pixels.

[0045] In this embodiment, the portrait generation optimization is performed to build a high-quality single-person oral video library. It is assumed that there is already a oral video library with a relatively complete data volume, such as the oral video library of Chan Mama; select videos with a single person, a resolution of 720p or above, and released in the past week; exclude videos with a face appearance time of less than 10s (which can be adjusted according to actual conditions). Taking the Chan Mama product category or the Douyin product category as an example, the video data under each product category is sorted in descending order according to the number of likes, and the top k videos (k is adjusted according to actual conditions) are taken and stored as high-quality oral videos of this category to obtain a high-quality single-person oral video library.

[0046] Build a high-quality voice-over character image feature library. For each product category, extract one video frame from k videos. The front face of the character must appear without any obstruction and the picture must be clear. The pictures all use a multimodal large model to summarize the characteristics of the photos (text description). One voice-over video has one character feature description, which will be used as a prompt for the character image, expressed as prompt.

[0047] Construct a diverse character image library, the multimodal model interacts with the image generation model multiple times, tunes the prompt, and generates character image pictures; the specific steps are: step a, take an image feature prompt, use the selected text image model to generate a character image; step b, the multimodal model determines the image quality and the difference between the image content and the prompt. If the quality does not pass, optimize the prompt, repeat steps a and b, and save the character image if it passes; step c, the number of times each prompt repeats steps a and b is set to a maximum of 10 times. If none of them pass, the prompt is discarded; stop when 5 are obtained (the parameters are adjustable); step d, get a preliminary character image library, each character in the library is a different character, there will be no copyright issues, and each short video can construct 0-5 generated new character images. Subsequently, generate images of different styles for each character and store them in the library; manually screen and filter to ensure quality twice.

[0048] Specifically in this embodiment, the following scheme is used to generate images of different styles for each character:

[0049] Generate portrait materials; first obtain the personal photos provided by the user, then call the portrait generation model to generate more portraits. The portrait generation model can be a self-developed model, an open source model, or a paid API, and supports high-quality real-person portrait generation models with images + prompt raw images.

[0050] Specifically, in this embodiment, the step of generating diversified portraits includes:

[0051] The first step is prompt design: manually design prompts, and after multiple experiments and optimization, summarize experience; based on experience, use LLM to generate a batch of prompts that meet the requirements of oral broadcast character images. Among them, the characteristics of characters in high-quality oral broadcast videos, taking fashion and clothing type short videos as an example, include: 1. Frontal face, high definition (1080P or above), 2. The proportion of the character to the screen (full-length image: about 1 / 5; half-length image: 1 / 5~1 / 3; close-up: 1 / 3~1 / 2), 3. Fresh and natural makeup, 4. Fashionable and generous clothes, avoid being too fancy or monotonous, 5. Look more friendly, natural, smiling and confident, 6. Appropriate expression of posture, 7. Background: different lighting, indoor and outdoor, background decoration (soft lighting, relatively clean background). Among them, in this embodiment, a prompt case is given: generate a fashionable female image with a confident smile, fresh and natural makeup, fashionable casual clothes, coordinated colors, and simple earrings and watches. The background is a modern interior scene, including simple furniture and green plants. The lighting is soft and the background is slightly blurred to highlight the characters.

[0052] The second step is to generate diverse portraits: obtain variables, which can be: different screen ratio requirements, different backgrounds, different clothing types and colors, different makeup, and character postures. After the prompt is determined, add a combination of one or more of the above variables to ensure that each prompt is different and generate diverse portraits.

[0053] The third step is screening: you can ask for prompts that can generate high-quality images and manually screen for prompts with high success rates; you can also use a multimodal model to screen, first tell the multimodal large model the requirements for high-quality portraits, and let the large model judge whether the current picture meets the requirements. The basis for judgment is: refer to the characteristics of the characters in the above high-quality oral videos (taking fashion and clothing type short videos as an example). Let the large model draw conclusions based on the judgment basis; for dimensions such as front face, clarity, and the proportion of characters in the screen, you can also use small models to verify: front face judgment-two-classification model; clarity judgment-two-classification model; character proportion: portrait segmentation, calculate the proportion of character pixels (error within 5% can be tolerated). Record the prompts corresponding to the images that the large model thinks are reasonable, count the success rate of each prompt, and keep the prompts with high success rates. Among them, the success rate of the prompt is determined according to the specific situation (the reference value of the success rate is 70%).

[0054] S102, extracting and processing from a preset oral broadcast video library, and inputting the extracted pictures and text information with characters into a multimodal large model for summarization to generate an oral broadcast video;

[0055] Specifically, step S102 includes: extracting a short video from a preset oral video library, and extracting the text information of the video, marked as T;

[0056] Randomly select 10 frames with people, stitch them together, and stitch them into a picture, marked as P1;

[0057] Input the image P1 and the text information T into the multimodal large model, summarize them, obtain the characteristics of the video and the behavior of the characters, and create a prompt word prompt corresponding to the short video;

[0058] Extracting a character image P2 of the same commodity category from the character image library;

[0059] Use the video generation model to process the character image P2 and the prompt word prompt to generate a video V;

[0060] 10 frames are randomly selected from the video V, the quality of each picture is judged using a multimodal large model, and videos that meet the quality requirements are stored in the database.

[0061] In this embodiment, the generation of spoken video is automatically optimized. From the constructed spoken video library, a short video is taken, 10 frames with characters are randomly selected, and they are spliced ​​into a picture, denoted as P1, and the text information of the video is extracted, denoted as T; P1 and T are input into the multimodal big model, and the big model is allowed to summarize the characteristics of the video and the behavior of the characters, and create a prompt for generating a spoken short video; the character image P2 of the same commodity category is taken from the character image library, and a video V is generated by the video generation model in combination with the prompt. Ten frames of video V are randomly selected, and the quality of each picture (clarity, character integrity, and similarity between the face and P2) is judged using the multimodal big model, and the videos that meet the quality requirements are put into the library.

[0062] S103: When it is determined that the generated spoken video has an abnormal problem, the spoken video is optimized for the abnormal problem.

[0063] Specifically, step S103 includes: extracting frames from the oral video to determine different angles of a face in the video;

[0064] When it is determined that the face in the video is not a frontal face, a frame is extracted and the extracted frame image is matched with the face to determine whether it is the same person;

[0065] If so, extract the side face frame with the highest similarity, call the video generation model to generate the next video segment based on the side face frame, and its prompt word prompt remains unchanged.

[0066] Preferably, the method further specifically comprises: using a 3D model to generate a side face image based on the front face video, wherein the side face angle is randomly selected within a preset range;

[0067] Generate a video based on a frontal face image, and add a reminder in the prompt that the facial shaking amplitude is less than 10°;

[0068] Extract the side face image to generate a video, and prompt the model that the facial shaking amplitude is less than 5°.

[0069] In this embodiment, the abnormal problem of spoken video is optimized. In actual use, the side face of a person is suitable for easily generating a face that is inconsistent with the person himself. Two solutions are provided here to optimize this problem.

[0070] Solution 1: Extract frames to determine different angles of the face. For scenes where the face is not frontal, first extract frames to match with the frontal face to determine whether the person is alone. Take the side face frame with the highest similarity, and let the video generation model generate the next video clip based on the frame. The prompt word remains unchanged.

[0071] Solution 2: Based on the frontal face video, use the 3D model to generate a side face (the angle can be randomly selected within the set range); the frontal face picture is suitable for generating a video, and a prompt is added to indicate that the facial shaking amplitude is less than 10°; take the side face picture, generate a video, and prompt the model that the facial shaking amplitude is less than 5°; this solution can be used to generate video storyboards of people with different facial angles.

[0072] In summary, the diversity optimization and automated optimization method of the voice-over video can be combined with the business to batch generate videos that meet business requirements. At the same time, the generated diverse portraits and voice-over videos are optimized, and the time and economic costs of manual shooting and production throughout the entire process are reduced.

[0073] See also Figure 2 The second embodiment of the present invention further provides a device for optimizing the diversity and automation of oral video, which includes:

[0074] The diversified portrait optimization unit 201 is used to obtain personal photos provided by users, respectively construct a high-quality single-person oral video library, a high-quality oral character image feature library and a diversified character image library, and optimize the generation of portraits;

[0075] The spoken video optimization unit 202 is used to extract and process from a preset spoken video library, and input the extracted pictures with characters and text information into the multimodal large model for summarization to generate a spoken video;

[0076] The abnormal problem optimization unit 203 is used to optimize the abnormal problem of the spoken video when it is determined that the generated spoken video has an abnormal problem.

[0077] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for optimizing the diversity and automation of spoken video, characterized in that: include: Obtain personal photos provided by users, build a high-quality single-person oral video library, a high-quality oral character image feature library, and a diversified character image library, and optimize the generation of portraits; Extract and process the pre-set spoken video library, and input the extracted pictures and text information with characters into the multi-modal large model for summarization to generate spoken video; When it is determined that the generated spoken-word video has abnormal problems, the spoken-word video is optimized for the abnormal problems.

2. A method for diversity optimization and automatic optimization of oral video according to claim 1, characterized in that: Obtain personal photos provided by users, build a high-quality single-person oral video library, a high-quality oral character image feature library, and a diversified character image library, and optimize the generation of portraits. Specifically: Construct a high-quality single-person oral video library, in which the video data under each product category is sorted in descending order according to the number of likes, and the top k videos are taken and stored as the high-quality oral videos of this category, thereby obtaining a high-quality single-person oral video library; Construct a high-quality oral broadcast character image feature library, where multiple videos of each product category are extracted, and one video frame is extracted. The extracted images are summarized using a multimodal large model. One oral broadcast video has one character feature description, which will be used as a character image prompt, expressed as prompt, to obtain a high-quality oral broadcast character image feature library; Build a diverse character image library, which uses a multimodal large model to interact with the image generation model multiple times, tune the prompt word, generate character image pictures, pre-process each character, generate images of different styles, store them in the library, and perform screening and filtering to achieve secondary quality assurance.

3. The method for diversity optimization and automatic optimization of oral video according to claim 2, characterized in that: Use the multimodal large model to interact with the image generation model multiple times, tune the prompt word prompt, and generate character image pictures, specifically: Extract a prompt word prompt, use the preset text-based graph model to process the prompt word prompt, and generate a character image; Use a large multimodal model to judge the quality of the image and the difference between the image content and the prompt word; When it is determined that the quality of the image does not meet the preset conditions, the prompt word is optimized, a prompt word is re-extracted, the prompt word is processed using the preset text-based image model, and a character image is generated for judgment; When it is determined that the quality of the image meets the preset conditions, the image of the person is saved; Each prompt can only be repeated 10 times at most. If the person still does not meet the preset conditions, the prompt will be discarded. When it is determined that a total of 5 character images are saved, stop and obtain a preliminary character image library, in which each character in the library is a different character.

4. The method for diversity optimization and automatic optimization of oral video according to claim 1, characterized in that: Preprocess each character to generate images of different styles, specifically: Obtain prompt design experience, wherein the prompt design experience is obtained by summarizing the initial prompt after multiple experimental adjustments; Using the LLM model to process according to the prompt design experience, multiple prompts that meet the requirements of the preset spoken character images are generated; Obtaining variable information, wherein the variable information includes different screen ratio requirements, different backgrounds, different clothing types and colors, different makeup, and different character postures; On the basis of multiple prompts that meet the requirements of the preset spoken character image, one or more combinations of variables in the variable information are added to ensure that the prompts in each combination are different; A multimodal large model is used to screen the prompts after adding variables, select the prompts with the highest success rate, and generate diverse portraits.

5. The method for diversity optimization and automatic optimization of oral video according to claim 4, characterized in that: The multimodal large model is used to screen the prompts after adding variables to select the prompts with the highest success rate, specifically: Obtain high-quality portrait requirements, call the multimodal large model to make a judgment based on the high-quality portrait requirements, and judge whether the current portrait image meets the high-quality portrait requirements; Record the prompt corresponding to the portrait image that meets the high-quality portrait requirements as determined by the multimodal large model; Count the success rate of each matching prompt and keep the prompt with the highest success rate.

6. A method for diversity optimization and automatic optimization of oral video according to claim 5, characterized in that: The multimodal large model is called to make a judgment according to the high-quality portrait requirement to determine whether the current portrait image meets the high-quality portrait requirement, specifically: The characteristics of the characters in the high-quality oral broadcast video are used as a judgment basis, and the multimodal large model makes a judgment based on the judgment basis; A small model is used to correct the front face, clarity, and the proportion of the person in the screen. Among them, a binary classification model is used to judge the front face, a binary classification model is used to judge the clarity, and a portrait segmentation model is used to calculate the proportion of the person's pixels.

7. The method for diversity optimization and automatic optimization of oral video according to claim 2, characterized in that: Extract and process the preset oral video library, and input the extracted pictures and text information with characters into the multimodal large model for summary to generate oral video. Specifically: Extract a short video from the preset oral video library and extract the video's copywriting information, marked as T; Randomly select 10 frames with people, stitch them together, and stitch them into a picture, marked as P1; Input the image P1 and the text information T into the multimodal large model, summarize them, obtain the characteristics of the video and the behavior of the characters, and create a prompt word prompt corresponding to the short video; Extracting a character image P2 of the same commodity category from the character image library; Use the video generation model to process the character image P2 and the prompt word prompt to generate a video V; 10 frames are randomly selected from the video V, the quality of each picture is judged using a multimodal large model, and videos that meet the quality requirements are stored in the database.

8. The method for diversity optimization and automatic optimization of oral video according to claim 1, characterized in that: When it is determined that the generated oral broadcast video has an abnormal problem, the oral broadcast video is optimized for the abnormal problem, specifically: Extract frames from the oral video to determine different angles of the face in the video; When it is determined that the face in the video is not a frontal face, a frame is extracted and the extracted frame image is matched with the face to determine whether it is the same person; If so, extract the side face frame with the highest similarity, call the video generation model to generate the next video segment based on the side face frame, and its prompt word prompt remains unchanged.

9. The method for diversity optimization and automatic optimization of oral video according to claim 1, characterized in that: When it is determined that the generated oral broadcast video has an abnormal problem, the oral broadcast video is optimized for the abnormal problem, specifically: Using the 3D model, a side face image is generated based on the frontal face video, and the side face angle is randomly selected within a preset range; Generate a video based on a frontal face image, and add a reminder in the prompt that the facial shaking amplitude is less than 10°; Extract the side face image to generate a video, and prompt the model that the facial shaking amplitude is less than 5°.

10. A device for optimizing diversity and automation of oral video, characterized in that: include: A diversified portrait optimization unit is used to obtain personal photos provided by users, and to construct a high-quality single-person oral video library, a high-quality oral character image feature library, and a diversified character image library, respectively, to optimize the generation of portraits; The spoken video optimization unit is used to extract and process the images and text information with characters from the preset spoken video library, and input the extracted images and text information with characters into the multimodal large model for summarization to generate the spoken video; The abnormal problem optimization unit is used to optimize the abnormal problem of the spoken video when it is determined that the generated spoken video has an abnormal problem.

Images