Batch generation method and device for directly published broadcast videos

Abstract

The invention provides a method and a device for batch generation of a directly published movie playing video, and the method comprises the steps: obtaining a to-be-processed video material, and marking the to-be-processed video material; performing copywriting extraction on industry hot videos to obtain video copywriting, searching for hotspots and golden 3s templates most related to the video copywriting, and performing imitation writing in combination with the hotspots of the video copywriting, the golden 3s templates and video features to obtain final imitation writing copywriting; according to the copywriting style and the semantic label of the video material, matching appropriate sound, and cloning the most appropriate sound by using a preset sound cloning model; and according to the most suitable sound, using a digital human model to carry out lip-shaped driving on the figure of the video material, generating subtitles, and obtaining an oral playing video corresponding to the video material. According to the method, for the generated video material, video content understanding, oral playing video copywriting generation, character sound matching suitable for the scene and digital human driving are combined, and oral playing videos which can be directly published are automatically generated in batches.

Description

Technical Field

[0001] The present invention relates to the technical field of spoken video generation, and in particular to a method and device for batch generating spoken videos that can be directly published. Background Art

[0002] The current solutions for generating task images with large models commonly used in the market are not realistic enough, and their sense of appeal and attraction do not always meet the requirements of use; specifically, the generated character images may not be fine enough in detail processing, and it is difficult to meet the high-value standards generally recognized by the public, and the clothing and makeup effects may also lack diversity and realism. At the same time, the angles of video / photo shooting and character movements are single. Most of the generated videos use fixed shooting angles, lack changes, and it is difficult to achieve real lens movement effects, such as push-pull, shaking, etc.; and the character makeup and clothing are single, and the emotional expression, action, background, and lighting effects are single. The realism and dynamic changes of the generated background are usually fake and the degree of integration is not high due to the background obtained by the method of cutting out the image to change the background. There are also copyright and privacy issues. This is because the frequent use of public videos on the Internet may involve risks of portrait rights and privacy rights; the works generated by changing the expressions and actions of characters in the video may have originality problems due to excessive similarity. Finally, this solution is difficult to implement technically and has high costs when used in practice.

[0003] Specifically, existing speech-by-video generation solutions mostly use manual shooting, which is costly; they use online video materials for editing without obtaining copyright; and most videos are generated using large models, using the portrait + prompt method to generate speech-by-video short video materials, which has unreliable results, false details, high difficulty, high failure rate, and uncontrollable quality. They also use body-driven / expression-driven solutions with fixed movements, fixed makeup, and fixed clothing, and single materials and lack of diversity.

[0004] In view of this, this application is filed. Summary of the invention

[0005] The present invention provides a method and device for batch generating spoken video that can be directly published, which can at least partially improve the above-mentioned problem.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for batch generating oral video that can be directly published, comprising:

[0008] Obtaining video material to be processed, performing labeling processing on the video material, and labeling the video material with semantic tags;

[0009] Extract the copy from the popular videos in the industry to obtain the video copy, search for the hot spots and golden 3s templates most relevant to the video copy, and imitate the video copy hot spots, golden 3s templates and video feature t1 to obtain the final imitation copy;

[0010] According to the text style and the semantic tags of the video material, a suitable voice is matched to obtain the most suitable voice, and the most suitable voice is cloned using a preset voice cloning model;

[0011] According to the most suitable voice, a digital human model is used to drive the lip movements of the characters in the video material, generate subtitles, and obtain a spoken video corresponding to the video material.

[0012] The present invention also provides a device for batch generating oral video that can be directly released, which comprises:

[0013] A video material understanding unit, used to obtain the video material to be processed, perform labeling on the video material, and label the video material with a semantic tag;

[0014] The copywriting unit is used to extract copywriting from popular videos in the industry to obtain video copywriting, search for hot spots and golden 3s templates most relevant to the video copywriting, and imitate the video copywriting hot spots, golden 3s templates and video feature t1 to obtain the final imitation copywriting;

[0015] A sound matching unit is used to match a suitable sound according to the text style and the semantic tag of the video material to obtain the most suitable sound, and clone the most suitable sound using a preset sound cloning model;

[0016] The digital human unit is used to drive the lip movements of the characters in the video material using the digital human model according to the most suitable voice, generate subtitles, and obtain the oral video corresponding to the video material.

[0017] In summary, the method for batch generating spoken-word videos that can be directly released proposes to combine video content understanding, spoken-word video text generation, character voice matching suitable for the scene, and digital human driving for the generated video materials, so as to automatically generate spoken-word videos that can be directly released in batches. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of a method for batch generating oral videos that can be directly published provided by the first embodiment of the present invention;

[0019] Figure 2 It is a module diagram of a device for batch generating spoken video that can be directly published provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0021] refer to Figure 1 As shown, the first embodiment of the present invention discloses a method for batch generating oral broadcast videos that can be directly published, which can be executed by a device for batch generating oral broadcast videos that can be directly published (hereinafter referred to as a generating device), and in particular, executed by one or more processors in the generating device to implement the following method:

[0022] S101, obtaining a video material to be processed, performing labeling processing on the video material, and labeling the video material with a semantic label;

[0023] Specifically, step S101 includes: taking the gender, age, and shooting scene of the person as semantic tags of the person in the video;

[0024] Randomly select 5 frames of pictures for splicing, and use the large model to understand the content of the spliced ​​pictures, summarize the video features, and mark them as t1.

[0025] In this embodiment, video material understanding is performed to give semantic tags to video materials. Five frames are randomly selected and spliced, and the large model is used to understand the image content and summarize the video characteristics, which is recorded as t1.

[0026] S102, extracting text from popular videos in the industry to obtain video text, searching for hot spots and golden 3s templates most relevant to the video text, and combining the video text hot spots, golden 3s templates and video feature t1 to perform imitation writing to obtain the final imitation text;

[0027] Specifically, step S102 includes: screening out popular products in the industry, and selecting the short video with the highest sales volume corresponding to the product for text extraction processing to obtain video text;

[0028] Searching for the two hot spots most relevant to the video copy in the preset hot spot library;

[0029] Search the preset golden 3s script template library for the five golden 3s templates that are most relevant to the video copy;

[0030] Combining the video copy, the two most relevant hot spots, the five most relevant golden 3s templates, and the video feature t1, the product to be processed is imitated to obtain multiple imitation copy, where one golden 3s line template corresponds to one imitation copy;

[0031] Use the big model to select the most qualified imitation copy from multiple imitation copies as the final imitation copy, marked as t2.

[0032] Preferably, popular commodities in the industry are screened out, specifically: popular commodities in the industry with high sales volume and high commission are selected.

[0033] Preferably, popular commodities in the industry are screened out, specifically by obtaining expert portraits and historical behaviors of platform experts, making commodity recommendations based on the expert portraits and the history of platform experts, and selecting popular commodities in the industry.

[0034] Preferably, the two hot spots most relevant to the video copy are searched in a preset hot spot library, specifically:

[0035] Take the video copy as the reference copy, use the LLM model to summarize the main content of the reference copy into one sentence, and use it as the main idea;

[0036] The subject is used as the search content, and a semantic vector search is performed from the hotspot library to obtain the two hotspots that are most relevant to the video copy.

[0037] In this embodiment, copywriting is performed, and there are two solutions for extracting imitation copywriting: Solution 1: extracting the copywriting of low-fan popular videos in each industry; Solution 2: extracting the copywriting of popular videos in the industry based on the automated product selection solution.

[0038] The specific steps of Solution 2 are as follows: You can use simple automatic product selection: select products with high sales and high commissions, for example, use the product selection platform to sort by commission and select the top K products. You can also use recommended product selection: recommend products based on the image of the influencer and the historical behavior of the influencer on the platform. Extract the copy of the short video with the highest sales corresponding to the above selected products.

[0039] Search for the two most relevant hot topics, where the preset hot topic library is updated every hour; search for the five most relevant golden 3s templates, where the preset golden 3s script template library is updated every day. Combine the above search information with the characteristics of the material in t1 to imitate the product, and each golden 3s script template corresponds to an imitation copy.

[0040] Specifically, take a popular video, that is, the copy of the popular copy extracted from the above imitation copy as the reference copy; use LLM to summarize the main content of the reference copy into one sentence, record it as the main idea; use the main idea as the search content, search (semantic vector retrieval) the two most relevant hot spots from the hot spot library; take a golden 3s line template, and require to use the template as the beginning, imitate the reference copy to write a short video copy, and refer to two hot spots for creation. This copy has a good beginning, takes advantage of the hot spot, and imitates the creation of popular copy, which can quickly complete a relatively high-quality video copy. Finally, use the big model to select the best copy, recorded as t2.

[0041] S103, matching appropriate voices according to the text style and the semantic tags of the video material to obtain the most appropriate voice, and cloning the most appropriate voice using a preset voice cloning model;

[0042] Specifically, step S103 includes: extracting video character semantic tags and text content from a preset high-quality oral material library as text features;

[0043] The text-sound data pairs consisting of text features and the sound corresponding to the video are used as positive samples, and the combination of text features and random speech is used as negative samples to construct a data set;

[0044] Using the dataset to train a text-to-sound matching model;

[0045] Use the trained text-sound matching model to match the most suitable sound for the text from the preset sound library;

[0046] The voice cloning model is used to clone the most suitable voice and the generated text.

[0047] Preferably, a text-sound matching model is trained using the data set, specifically:

[0048] Obtain a basic matching model, wherein the basic matching model is a twin network or a clip multimodal model;

[0049] The character semantic labels and text content in the data set are used as input text data of the basic matching model, and a 3s to 10s audio in the data set is used as input voice data of the basic matching model for training to obtain a model output, wherein the model output is 0 or 1, 0 indicates irrelevant, and 1 indicates relevant;

[0050] The training steps are repeated until the basic matching model reaches a preset standard, thereby obtaining a text-sound matching model.

[0051] Preferably, a trained text-sound matching model is used to match the sound that best matches the text from a preset timbre library, specifically:

[0052] Extracting the semantic tags of the characters and the text content of the current video, wherein the semantic tags of the characters in the video are extracted by the multimodal model according to the specified character semantic tag system;

[0053] Match the semantic tags of the characters in the video and the text content with all the sounds in the sound library, take the sounds with a correlation higher than 80%, and randomly select one of them to generate the most suitable sound for the text.

[0054] In this embodiment, appropriate sound is matched according to the text style and the semantic tags of the video material. The sound matching steps are as follows:

[0055] In the first step, we use the high-quality spoken-word material library constructed above to extract semantic labels of video characters [such as age, gender, etc.] and text content as text features, and use the sound corresponding to the video to form a text-sound data pair (select spoken-word videos with no background sound, no soundtrack, and no noise) as positive samples, and the combination of text and random speech as negative samples to complete the data set construction.

[0056] The second step is to use the data constructed above to train a text-sound matching model. Specifically: select a matching model, such as a twin network, or a multimodal model similar to clip; the model contains two modal data inputs, text (character semantic labels + copy content) and voice (a 3s to 10s audio); the model output is 0 (irrelevant) or 1 (relevant).

[0057] The third step is to use the above model to match the most suitable sound (assuming that there is a sound library, or so-called timbre library). Specifically, take the character semantic tags and text content of the current video. The semantic tags can be extracted by the multimodal model according to the specified character semantic tag system; match the character semantic tags and text content with all the sounds in the sound library, take the sounds with a correlation higher than 80%, and randomly select one of the sounds.

[0058] Finally, use the emotional voice cloning model to clone the above-mentioned matching voice and generated text (if you are not satisfied with the voice, you can also record your own voice to clone).

[0059] S104, according to the most suitable voice, use the digital human model to drive the lip movements of the characters in the video material, generate subtitles, and obtain the spoken video corresponding to the video material.

[0060] In this embodiment, a high-quality digital human model is used to drive the lip movements of the characters in the video material, and the input data is the sound data generated above. Subtitles are generated to obtain a video.

[0061] In summary, the method for batch generating spoken-word videos that can be directly released proposes to combine video content understanding, spoken-word video text generation, character voice matching suitable for the scene, and digital human driving for the generated video materials, so as to automatically generate spoken-word videos that can be directly released in batches.

[0062] See also Figure 2 The second embodiment of the present invention further provides a device for batch generating oral video that can be directly released, which includes:

[0063] The video material understanding unit 201 is used to obtain the video material to be processed, perform labeling on the video material, and label the video material with a semantic tag;

[0064] The copywriting unit 202 is used to extract the copywriting from the popular videos in the industry to obtain the video copywriting, search for the hot spots and golden 3s templates most relevant to the video copywriting, and imitate the video copywriting hot spots, golden 3s templates and video features t1 to obtain the final imitation copywriting;

[0065] The sound matching unit 203 is used to match the appropriate sound according to the text style and the semantic tag of the video material to obtain the most appropriate sound, and clone the most appropriate sound using a preset sound cloning model;

[0066] The digital human unit 204 is used to drive the lip movements of the characters in the video material using the digital human model according to the most suitable voice, generate subtitles, and obtain the spoken video corresponding to the video material.

[0067] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for batch generating oral videos that can be directly published, characterized in that: include: Obtaining video material to be processed, performing labeling processing on the video material, and labeling the video material with semantic tags; Extract the copy from the popular videos in the industry to obtain the video copy, search for the hot spots and golden 3s templates most relevant to the video copy, and imitate the video copy hot spots, golden 3s templates and video feature t1 to obtain the final imitation copy; According to the text style and the semantic tags of the video material, a suitable voice is matched to obtain the most suitable voice, and the most suitable voice is cloned using a preset voice cloning model; According to the most suitable voice, a digital human model is used to drive the lip movements of the characters in the video material, generate subtitles, and obtain a spoken video corresponding to the video material.

2. A method for batch generating oral videos that can be directly published according to claim 1, characterized in that: The video material is labeled and semantic tags are added to the video material, specifically: The gender, age, and shooting scene of the characters are used as semantic labels of the characters in the video; Randomly select 5 frames of pictures for splicing, and use the large model to understand the content of the spliced ​​pictures, summarize the video features, and mark them as t1.

3. A method for batch generating oral videos that can be directly published according to claim 2, characterized in that: Extract the copywriting of the popular videos in the industry to obtain the video copywriting, search for the hot spots and golden 3s templates most relevant to the video copywriting, and imitate the video copywriting hot spots, golden 3s templates and video features t1 to obtain the final imitation copywriting, which is specifically: Filter out the popular products in the industry, and select the short videos with the highest sales volume corresponding to the products for copy extraction and processing to obtain video copy; Searching for the two hot spots most relevant to the video copy in the preset hot spot library; Search the preset golden 3s script template library for the five golden 3s templates that are most relevant to the video copy; Combining the video copy, the two most relevant hot spots, the five most relevant golden 3s templates, and the video feature t1, the product to be processed is imitated to obtain multiple imitation copy, where one golden 3s line template corresponds to one imitation copy; Use the big model to select the most qualified imitation copy from multiple imitation copies as the final imitation copy, marked as t2.

4. The method for batch generating oral videos that can be directly published according to claim 3 is characterized in that: Filter out the popular products in the industry, specifically: Choose popular products in the industry with high sales and high commissions.

5. The method for batch generating oral videos that can be directly published according to claim 3 is characterized in that: Filter out the popular products in the industry, specifically: Obtain the expert portrait and the historical behavior of the platform expert, make product recommendations based on the expert portrait and the platform expert's history, and select popular products in the industry.

6. The method for batch generating oral videos that can be directly published according to claim 3 is characterized in that: Search the preset hotspot library for the two hotspots most relevant to the video copy, specifically: Take the video copy as the reference copy, use the LLM model to summarize the main content of the reference copy into one sentence, and use it as the main idea; The subject is used as the search content, and a semantic vector search is performed from the hotspot library to obtain the two hotspots that are most relevant to the video copy.

7. The method for batch generating oral videos that can be directly published according to claim 1, characterized in that: According to the copywriting style and the semantic tags of the video material, the appropriate sound is matched to obtain the most suitable sound, and the most suitable sound is cloned using a preset sound cloning model, specifically: Extract the semantic labels of video characters and text content from the preset high-quality oral material library as text features; The text-sound data pairs consisting of text features and the sound corresponding to the video are used as positive samples, and the combination of text features and random speech is used as negative samples to construct a data set; Using the dataset to train a text-to-sound matching model; Use the trained text-sound matching model to match the most suitable sound for the text from the preset sound library; The voice cloning model is used to clone the most suitable voice and the generated text.

8. The method for batch generating oral videos that can be directly published according to claim 7, characterized in that: Use the dataset to train a text-to-sound matching model, specifically: Obtain a basic matching model, wherein the basic matching model is a twin network or a clip multimodal model; The character semantic labels and text content in the data set are used as input text data of the basic matching model, and a 3s to 10s audio in the data set is used as input voice data of the basic matching model for training to obtain a model output, wherein the model output is 0 or 1, 0 indicates irrelevant, and 1 indicates relevant; The training steps are repeated until the basic matching model reaches a preset standard, thereby obtaining a text-sound matching model.

9. The method for batch generating oral videos that can be directly published according to claim 7, characterized in that: Use the trained text-sound matching model to match the most suitable sound for the text from the preset sound library, specifically: Extracting the semantic tags of the characters and the text content of the current video, wherein the semantic tags of the characters in the video are extracted by the multimodal model according to the specified character semantic tag system; Match the semantic tags of the characters in the video and the text content with all the sounds in the sound library, take the sounds with a correlation higher than 80%, and randomly select one of them to generate the most suitable sound for the text.

10. A device for batch generating oral videos that can be directly published, characterized in that: include: A video material understanding unit, used to obtain the video material to be processed, perform labeling on the video material, and label the video material with a semantic tag; The copywriting unit is used to extract copywriting from popular videos in the industry to obtain video copywriting, search for hot spots and golden 3s templates most relevant to the video copywriting, and imitate the video copywriting hot spots, golden 3s templates and video feature t1 to obtain the final imitation copywriting; A sound matching unit is used to match a suitable sound according to the text style and the semantic tag of the video material to obtain the most suitable sound, and clone the most suitable sound using a preset sound cloning model; The digital human unit is used to drive the lip movements of the characters in the video material using the digital human model according to the most suitable voice, generate subtitles, and obtain the oral video corresponding to the video material.

Images