AI short video production device and method based on large language model

Abstract

The invention provides an AI short video production device and method based on a large language model, the device comprises a copywriting rewriting module, a product implantation module, a digital population broadcast video generation module and a short video automatic editing module, and the copywriting rewriting module rewrites a population broadcast copywriting conforming to a client IP; the product implanting module forms core materials according to details of services or products which are expected by customers to be implanted into videos; the digital human mouth broadcast video generation module is combined with a mouth broadcast copywriting driver to output the mouth shape and expression of a digital human to form a video file; and the automatic short video editing module generates a final short video slice with subtitles and effects. According to the AI short video production device based on the large language model, the short video meeting the requirements and positioning of the customer can be quickly generated for the customer, and the zero-threshold use of short video creation is realized for the customer.

Description

Technical Field

[0001] The present invention relates to the field of AI application technology, and in particular to an AI short video production device and method based on a large language model. Background Art

[0002] Currently, in content marketing, especially in the field of short video marketing, traditional GUI (Graphic user interface) software tools are basically used to implement the topic selection, copywriting and post-shooting editing functions of short videos. The user entry threshold is very high, so it is basically through self-built content teams or entrusted third-party operation teams to continuously produce IP (Intellectual Property) short videos for customers. After the emergence of AI (Artificial Intelligence) large language model technology, because the semantic understanding and content production capabilities have reached the expert level of output, it is a future technology trend to train and use AI agents as real corporate employees. At present, with the rise of short video marketing, more and more business owners hope to achieve short video marketing and customer acquisition through IP exposure, and the output of such short videos through natural dialogue is the closest way for bosses to assign tasks to employees. However, there are no AI-native IP short video creation products yet.

[0003] After short video platforms became the main way for Internet users to obtain content, customers paid more and more attention to how to reach their users through short video platforms and influence these users to form commercial conversions. However, the entire chain of short video production includes planning, topic selection, copywriting, shooting, editing, operation and service. For small and medium-sized enterprises that lack market content teams, each link basically faces the challenges of lack of knowledge system, lack of time and energy, lack of funds, and lack of team. It is very difficult to combine them together. It is basically difficult to achieve high-frequency updates every day. If you use an external operation team, you will face high agency operation fees, and the company cannot precipitate the methodology of short video operation. In order to enable more companies to have their own content teams more conveniently, quickly and cheaply, we have invented a new AI short video production device based on a large language model. Summary of the invention

[0004] The purpose of the present invention is to provide an AI short video production device based on a large language model that can enable enterprises to have their own content team more conveniently, quickly and cheaply.

[0005] The purpose of the present invention can be achieved through the following technical measures: an AI short video production device based on a large language model, the AI ​​short video production device based on a large language model includes a text rewriting module, a product implantation module, a digital human voice-over video generation module and a short video automatic editing module, the text rewriting module rewrites the oral text that conforms to the customer's IP; the product implantation module forms core materials according to the details of the business or product that the customer wants to be implanted in the video; the digital human voice-over video generation module drives the output of the digital human's lip shape and expression in combination with the oral text to form a video file; the short video automatic editing module collects the video file output by the digital human voice-over video module, the text output by the text rewriting module and the core material output by the product implantation module to generate a short video with subtitles and effects as a final film.

[0006] The purpose of the present invention can also be achieved by the following technical measures:

[0007] The AI ​​short video production device also includes a link parsing module, which performs in-depth analysis of link content based on pre-prepared topics and links, including capturing text links and ASR analysis of video / audio links, to generate original materials.

[0008] The copy rewriting module is trained on the prompt words for the customer's business positioning, customer group positioning and personality positioning preset by the large language model, and then rewrites the oral copy that conforms to the customer's IP in combination with the AIGC capability of the large model; the product implantation module uses RAG technology and vector database technology to parse the customer's knowledge base documents, and embeds the details of the customer's business or product that the customer hopes to embed in the external video, forming some core materials to assist the customer in enhancing the explanation in the externally released short video.

[0009] The digital human voice-over video generation module is divided into a training phase and an inference phase. In the training phase, the digital human voice-over video generation module trains a video of the customer to form a digital human that is a 1:1 replica of the customer's image and voice; in the inference phase, the digital human voice-over video generation module will combine the output of the oral text to drive the output of the digital human's lip shape and expression to form a video file that plays the customer's oral broadcast in a 1:1 ratio.

[0010] The AI ​​short video production device also includes a short video poster generation module and an all-platform automatic publishing module. The short video poster cover generation module generates poster covers for short video platforms in three ways. The first is to add title text generated by AI summary based on the customer's uploaded photos. The second is to collect the first frame of the short video and add the title summary text generated by AI. The third is to generate a poster cover based on AI and match it with the video summary text generated by AI. After obtaining the initialization scan code authorization of the customer's account, the all-platform automatic publishing module automatically associates with the customer's multi-platform account, and automatically publishes the video on multiple platforms according to the platform selected by the customer before each release.

[0011] The purpose of the present invention can also be achieved by the following technical measures: an AI short video production method based on a large language model, the AI ​​short video production method based on a large language model includes:

[0012] Step 1: Collect the customer’s video and audio files for digital human pre-training;

[0013] Step 2: Generate oral text that meets the structural logic requirements through the text generation capability of the large language model;

[0014] Step 3: Combine the oral text to drive the output of the digital human's lip shape and expression to form a video file that plays the oral broadcast 1:1 for the customer;

[0015] Step 4: Generate a short video with subtitles and effects based on the spoken text, the spoken video file and the core materials;

[0016] Step 5: Generate a poster cover for the short video platform and publish it on the short video media platform across the entire network.

[0017] The purpose of the present invention can also be achieved by the following technical measures: Step 1 specifically includes:

[0018] Step 11, shooting a video of the customer speaking to the camera;

[0019] Step 12, collect the audio material of the customer and pre-train the digital human to train the timbre;

[0020] Step 13, providing the materials of steps 11 and 12 to the digital human training platform for training, so as to complete a 1:1 replica of the real person;

[0021] Step 14: Produce the content positioning of the short video based on the client’s personality, business, and user positioning, and input the prompt words into the large language model;

[0022] Step 15: Organize and upload the company's external marketing-oriented knowledge documents separately, and use the large language model to learn and interpret them to prepare for the subsequent generation of short video materials.

[0023] Step 2 specifically includes:

[0024] Step 21, rewrite the copy by using the text generation capability of the large language model and combining the content of the preset customer personality and business positioning prompt words;

[0025] Step 22, through RAG search enhancement generation technology, the details of related businesses or products in the customer knowledge base are found out, forming graphic materials for product implantation, and forming a corresponding relationship between the pictures and the spoken text;

[0026] Step 23, reorganize the text produced in steps 21 and 22 to generate a set of text that meets the structural logic requirements.

[0027] Step 4 specifically includes:

[0028] Step 41, based on the generated copy and the generated oral video, a video with IP image and core materials is formed;

[0029] In step 42, the large language model automatically completes sentence segmentation and pauses based on the semantic understanding of the text, calls the default or customer-preset subtitle style, generates subtitles, and highlights some key subtitles.

[0030] Step 5 specifically includes:

[0031] Step 51, generating a title, topic and cover of the video to be released according to the copy;

[0032] Step 52, obtain the publishing permission based on the platform authorized by the customer. When each video work is released, the customer selects the platform on which he / she publishes it, and publishes the short video to the global mainstream short video platform through interface calls.

[0033] The AI ​​short video production device based on a large language model in the present invention helps customers build content production teams through intelligent agent application technology driven by a large language model, opening up a new employment paradigm that provides exclusive short video production services to customers 24 hours a day, 7 days a week. The present invention can quickly generate short videos that meet customer requirements and positioning for customers through an AI short video production device based on a large language model, allowing customers to use short video creation with zero threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a structural diagram of a specific embodiment of an AI short video production device based on a large language model of the present invention;

[0035] Figure 2 It is a flowchart of a specific embodiment of the AI ​​short video production method based on a large language model of the present invention;

[0036] Figure 3 A flowchart of digital human training in a specific embodiment of the present invention;

[0037] Figure 4 A flowchart of copywriting material generation in a specific embodiment of the present invention;

[0038] Figure 5 The present invention is a flowchart of video generation and publishing in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations and / or combinations thereof.

[0041] The AI ​​short video production device based on a large language model of the present invention utilizes prompt word engineering technology in combination with the customer's business positioning and user positioning, as well as the oral style, to generate a video script with the customer's unique personality style, and combines the digital human reproduction technology and intelligent editing technology of the customer's exclusive image to generate a complete customer IP short video, which is automatically published on the short video media platform of the entire network.

[0042] Figure 1 This is a module diagram of the AI ​​short video production device based on the large language model of the present invention. The AI ​​short video production device includes seven modules: a link parsing module 21, a text rewriting module 22, a product implantation module 23, a digital voice broadcast video generation module 24, a short video automatic editing module 25, a short video poster generation module 26 and a full-platform automatic publishing module 27.

[0043] The link analysis module 21 performs in-depth analysis of the link content based on the received selected topics and captured links, including capturing text links and ASR (audio to text analysis) analysis of video / audio links, so as to generate original materials.

[0044] After training the prompt words of the customer's business positioning (business, business model), customer positioning (customer portrait) and personality positioning (personality, speaking style) preset by the large language model, the copy rewriting module 22 can combine the AIGC (Artificial Intelligence Generated Content) capability of the large model to rewrite the oral copy that conforms to the customer's IP.

[0045] The product implantation module 23 refers to the use of RAG (retrieval enhanced generation) technology and vector database technology to parse the company's knowledge base documents (pdf, jpg, word, ppt, etc.), and to embed the details of the company's business or products (such as highlights, cases, brands, comparison charts) in external videos, so as to form some core materials to assist customers in enhancing explanations in short videos released to the outside world. For example, when talking about the comparison of the number of immigrants to the United States, a trend chart of the number of Chinese immigrants to the United States in the past 10 years and the number of people of various identities in the company's knowledge base can be presented in detail in the lower middle part of the video.

[0046] The digital human voice broadcast video generation module 24 is divided into a training phase and an inference phase. In the training phase, only a 1-3 minute video of the customer is needed for training to form a digital human that is a 1:1 replica of the customer's image and voice. In the inference phase, the digital human voice broadcast module will combine the output of the oral broadcast copy to drive the output of the digital human's lip shape and expression to form a video file that plays the customer's oral broadcast in a 1:1 ratio.

[0047] The short video automatic editing module 25 can automatically collect the video output of the digital voice broadcast video module, the text output of the text rewriting module and the output of the patch material of the product placement module, and can automatically generate a short video with subtitles and effects as a final film.

[0048] The short video poster cover generation module 26 can generate poster covers for short video platforms in three ways. The first is to automatically add title text generated by AI summary based on the photos uploaded by the customer. The second is to automatically collect the first frame of the short video and add the title summary text generated by AI. The third is to generate the poster cover completely based on AI and match it with the video summary text generated by AI.

[0049] The full-platform automatic publishing module 27 can automatically link the customer's multi-platform accounts (Tik Tok, Kuaishou, Video Account, Xiaohongshu, etc.) after obtaining the initial scan code authorization of the customer's account. Before each release, the video will be automatically published on multiple platforms according to the platform selected by the customer, and the customer will not be aware of the entire publishing process.

[0050] Figure 2The flowchart of the AI ​​short video production method based on the large language model of the present invention is as follows. The AI ​​short video production method based on the large language model specifically includes:

[0051] Step 101, collecting the client's video and audio files for digital human pre-training;

[0052] Step 102, using the text generation capability of the large language model to generate a spoken text that meets the structural logic requirements;

[0053] Step 103, combining the output oral text to drive the output of the digital human's mouth shape and expression, forming a video file for the customer to play the oral broadcast in a 1:1 ratio;

[0054] Step 104, generating a short video with subtitles and effects as a final film according to the spoken text, the video file of the spoken broadcast, and the core materials;

[0055] Step 105, generate a poster cover for the short video platform and publish it on the short video media platform of the entire network.

[0056] Figure 3 This is a specific flow chart of pre-training digital humans in the AI ​​short video production method based on a large language model of the present invention.

[0057] Step S1011, shoot a 1-3 minute video of the customer in a certain scene. For example, according to the theory that people, goods and places are consistent in short videos, you should choose a scene that is suitable for the customer to sell goods and shoot a 1-3 minute video. For example, for immigration agencies, the best scene is to shoot a 1-3 minute video in their own office or conference room with neatly arranged bookcases and archives behind, and the customer faces the camera to make a 1-3 minute video.

[0058] In step S1012, in order to achieve better sound restoration, 1-3 minutes of audio material from the customer can be collected separately to pre-train the digital human, mainly training the timbre.

[0059] In step S1013, the materials of the previous two steps are uniformly provided to the digital human training platform for training to complete a 1:1 replica of a real person.

[0060] In step S1014, the content positioning of the short video is generated based on the personality, business and user positioning completed face-to-face with customers offline, and these inputs are given to the prompt words of the large language model, so that the AI ​​device can better tailor its services to customers.

[0061] In step S1015, in order to better integrate with the client’s own business, we also need to separately organize and upload the client’s external marketing-oriented knowledge documents, and use the large language model to learn and interpret them, so as to prepare for the subsequent generation of short video materials.

[0062] Figure 4 This is a flowchart of copywriting material generation in a specific embodiment of the present invention.

[0063] In step S1031, through data communication, the AI ​​short video production device already knows that a hot topic about changes in US immigration policy will be released at 9 o'clock this morning, and goes online to start work on time at 9 o'clock.

[0064] In step S1032, the copy rewriting module uses the text generation capability of the large language model and combines it with the content of the preset customer personality and business positioning prompts to automatically rewrite the copy. For example, it will emphasize that the author is an immigration expert with 10 years of service in the industry and has never failed in any case, and the language style will be close to the customer's personality.

[0065] In step S1033, the product placement module uses RAG search enhancement generation technology to find all the cases and service highlights related to investment immigration in the enterprise knowledge base, and forms graphic materials for product placement, as well as the corresponding relationship between pictures and spoken text. For example, when our investment immigration helped the boss of Enterprise A get his identity card in as fast as 50 days, it corresponded to a photo of the boss of Enterprise A thanking the company after he got his identity card.

[0066] In step S1034, the system will reorganize the documents produced in the above two steps to generate a set of documents that meet the structural logic requirements.

[0067] In step S1035, because all short video platforms require the upload of a cover image when publishing a video, we use the summarization and text-based image capabilities of the large language model to help customers generate the cover image required for each short video. For example, the base image is still an image of the IP of the owner of an immigration agency, and the title above reads how investment immigrants can obtain status within 2 months.

[0068] Figure 5 The present invention is a flowchart of video generation and publishing in a specific embodiment of the present invention.

[0069] In step S1041, using the digital human voice broadcast video generation module, all oral texts will be driven by TTS technology (Text To Speech) to drive the trained customer digital human to make oral broadcast videos. For example, if there is a text of about one minute with 300 words generated, then the digital human can broadcast this one-minute oral video.

[0070] In step S1042, because the customer wants to complete more effective information transmission in the short video, the text materials of the product and service categories generated in S1033 above will be uniformly loaded into the oral video generated by S1041 through the short video automatic editing module at this time, forming a video with an IP image and material patches.

[0071] In step S1043, the large language model will automatically complete sentence breaks and pauses based on the semantic understanding of the text, call the system default or customer preset subtitle style, generate subtitles, and highlight some key subtitles, such as the three pitfalls of investment immigration to emphasize such information.

[0072] In step S1044, the short video poster generation module will automatically generate the title, topic and cover of the video according to the copy, and pass the completed cover to the full platform automatic publishing module. For example, the cover will be equipped with the client's IP image photo and the large words "The US immigration law has changed dramatically" and the small words "You can get investment immigration in 30 days".

[0073] In step S1045, the full-platform automatic publishing module will obtain publishing permissions based on the platforms authorized by the customer. When each video work is released, the customer only needs to check the platform where he or she publishes it (such as TikTok, Video Account, and Kuaishou), and publish the short video to the world's mainstream short video platforms through interface calls.

[0074] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0075] Except for the technical features described in the specification, all other technical features are known technologies to those skilled in the art.

Claims

1. An AI short video production device based on a large language model, characterized in that: The AI ​​short video production device based on the large language model includes a copy rewriting module, a product implantation module, a digital human voice-over video generation module and a short video automatic editing module. The copy rewriting module rewrites the oral copy that conforms to the customer's IP; the product implantation module forms core materials according to the details of the business or product that the customer wants to be implanted in the video; the digital human voice-over video generation module drives the output of the digital human's lip shape and expression in combination with the oral copy to form a video file; the short video automatic editing module collects the video file output by the digital human voice-over video module, the copy output by the copy rewriting module and the core materials output by the product implantation module to generate a short video with subtitles and effects as a final film.

2. The AI ​​short video production device based on a large language model according to claim 1, characterized in that: The AI ​​short video production device also includes a link parsing module, which performs in-depth analysis of link content based on pre-prepared topics and links, including capturing text links and ASR analysis of video / audio links, to generate original materials.

3. The AI ​​short video production device based on a large language model according to claim 1, characterized in that: The copy rewriting module is trained on the prompt words for the customer's business positioning, customer group positioning and personality positioning preset by the large language model, and then rewrites the oral copy that conforms to the customer's IP in combination with the AIGC capability of the large model; the product implantation module uses RAG technology and vector database technology to parse the customer's knowledge base documents, and embeds the details of the customer's business or product that the customer hopes to embed in the external video, forming some core materials to assist the customer in enhancing the explanation in the externally released short video.

4. The AI ​​short video production device based on a large language model according to claim 1, characterized in that: The digital person broadcasting video generation module is divided into a training phase and an inference phase. In the training phase, the digital person broadcasting video generation module trains a video of the customer to form a digital person that is a 1:1 replica of the customer's image and voice. During the inference stage, the digital human voice-over video generation module will combine the output of the voice-over text to drive the output of the digital human's lip shape and expression, forming a 1:1 voice-over video file for the customer.

5. The AI ​​short video production device based on a large language model according to claim 1, characterized in that: The AI ​​short video production device also includes a short video poster generation module and an all-platform automatic publishing module. The short video poster cover generation module generates poster covers for short video platforms in three ways. The first is to add title text generated by AI summary based on the customer's uploaded photos. The second is to collect the first frame of the short video and add the title summary text generated by AI. The third is to generate a poster cover based on AI and match it with the video summary text generated by AI. After obtaining the initialization scan code authorization of the customer's account, the all-platform automatic publishing module automatically associates with the customer's multi-platform account, and automatically publishes the video on multiple platforms according to the platform selected by the customer before each release.

6. The AI ​​short video production method based on a large language model is characterized by: The AI ​​short video production method based on the large language model includes: Step 1: Collect the customer’s video and audio files for digital human pre-training; Step 2: Generate oral copy that meets the structural logic requirements through the text generation capability of the large language model; Step 3: Combine the oral text to drive the output of the digital human's lip shape and expression to form a video file that plays the oral broadcast 1:1 for the customer; Step 4: Generate a short video with subtitles and effects based on the spoken text, the spoken video file and the core materials; Step 5: Generate a poster cover for the short video platform and publish it on the short video media platform across the entire network.

7. The AI ​​short video production method based on a large language model according to claim 6, characterized in that: Step 1 specifically includes: Step 11, shooting a video of the customer speaking to the camera; Step 12, collect the audio material of the customer and pre-train the digital human to train the timbre; Step 13, providing the materials of steps 11 and 12 to the digital human training platform for training, so as to complete a 1:1 replica of the real person; Step 14: Produce the content positioning of the short video based on the client’s personality, business, and user positioning, and input the prompt words into the large language model; Step 15: Organize and upload the company's external marketing-oriented knowledge documents separately, and use the large language model to learn and interpret them to prepare for the subsequent generation of short video materials.

8. The AI ​​short video production method based on a large language model according to claim 6, characterized in that: Step 2 specifically includes: Step 21, rewrite the copy by using the text generation capability of the large language model and combining the content of the preset customer personality and business positioning prompt words; Step 22, through RAG search enhancement generation technology, the details of related businesses or products in the customer knowledge base are found out, forming graphic materials for product implantation, and forming a corresponding relationship between the pictures and the spoken text; Step 23, reorganize the text produced in steps 21 and 22 to generate a set of text that meets the structural logic requirements.

9. The AI ​​short video production method based on a large language model according to claim 6, characterized in that: Step 4 specifically includes: Step 41, based on the generated copy and the generated oral video, a video with IP image and core materials is formed; In step 42, the large language model automatically completes sentence segmentation and pauses based on the semantic understanding of the text, calls the default or customer-preset subtitle style, generates subtitles, and highlights some key subtitles.

10. The AI ​​short video production method based on a large language model according to claim 6, characterized in that: Step 5 specifically includes: Step 51, generating a title, topic and cover of the video to be released according to the copy; Step 52, obtain the publishing permission based on the platform authorized by the customer. When each video work is released, the customer selects the platform on which he / she publishes it, and publishes the short video to the global mainstream short video platform through interface calls.

Images