A method for implementing a text-driven video generation system

Abstract

The invention discloses a method for realizing a text-driven video generation system, which integrates a speech synthesis system and a video generation system into one system. The speech synthesis system adopts a JSP page of dynamic Web technology, completes the generation of speech by executing a speech synthesis model trained by a back-end server, and then transmits the speech to the front end through a framework, wherein modules such as adding, deleting, querying, batch deleting, synthesized speech downloading, and synthesized speech speed change are added. The video generation system takes the speech synthesis system as the cornerstone, and can adopt speech generated by the speech synthesis system, speech uploaded by any user, and any face picture, and transmits the speech and picture to the back-end server through a JSP page, and the back end loads the trained video generation model to complete the generation of the video, and then transmits the video to the front end through the framework, thereby realizing a complete text-driven video generation system.

Description

Technical Field

[0001] The present invention relates to an implementation method of a text-driven video generation system, and belongs to the technical fields of artificial intelligence, deep learning, and speech processing. Background Art

[0002] Most of the existing models currently only focus on speech synthesis or video generation. For example, for speech synthesis, most researchers focus on model tuning. From Melgan to Tacoron, to Fast-Speech2, these models can synthesize relatively realistic audio, but speech synthesis has not yet been integrated into the system. In addition, in the field of video generation, most researchers also only focus on model tuning. The existing GAN model and Wav2Lip model can also generate realistic videos, but video generation has not yet been integrated into the system.

[0003] For those who want to use speech synthesis or video generation, it is necessary to develop a text-driven video generation system that integrates speech synthesis and video generation.

[0004] In summary, for the development of text-driven video generation system, choosing to use the Fast-Speech2 model as the back-end service of the speech synthesis model can ensure that the generated speech is more realistic. Choosing to use the Wav2Lip model as the back-end service of the video generation model can ensure that the generated video is closer to the real scene. This will provide a better user experience for using the system and solve the current lack of speech synthesis systems and video generation systems. Summary of the invention

[0005] The purpose of the present invention is to provide a method for realizing a text-driven video generation system to solve the problem that there is no speech synthesis system and video generation system in the existing scenario introduced in the background technology. The method can utilize the existing high-performance Fast-Speech2 speech synthesis model and Wav2Lip video generation model, as well as the front-end language to complete the text-driven video generation system.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] The present invention provides a method for implementing a text-driven video generation system, and the specific steps are as follows:

[0008] Step 1: User login module design,

[0009] 1.1, create a MySQL database to store usernames and passwords;

[0010] 1.2, insert the username and password into the database;

[0011] Step 2: Front-end design of text-driven video generation system.

[0012] 2.1, select any picture that the user is interested in as the core picture displayed on the front-end interface;

[0013] 2.2, Use JavaScript language to build the front-end interface and insert the above pictures into the front-end page;

[0014] 2.3. Based on 2.2, the front-end language is used to implement the adding, deleting, querying, batch deleting, synthesized speech downloading, synthesized speech speed change module of the speech synthesis system, as well as the adding, deleting, querying, batch deleting, generating video downloading, and generating video speed change module of the video generation system.

[0015] 2.4, the interaction with the backend is completed by using the http protocol combined with the JSON data format. Two interfaces are mainly designed: speech synthesis interface and video generation interface. The speech synthesis interface is any API containing the output file name and input text, such as any API address in the format of " / api / text?filename={filename}&text={text}", where filename and text represent the voice file name and text content respectively. The request method adopts Get, and the returned data format is designed in Json format, including the status code and the returned message. The video generation interface is any API containing the video file name, voice type, voice file name and picture file name, such as the format of " / api / video?mp4name={mp4name}&checkpointname={checkpointname}&audioname={audiona me}&picname={picname}", where mp4name, checkpointname, audioname and picname represent the video file name, voice type, voice file name and picture file name respectively. The request method adopts Get, and the returned data format is designed in JSON format, including the status code and the returned message;

[0016] Step 3: Design of backend service for speech synthesis system.

[0017] 3.1, collect the Chinese speech synthesis dataset AISHELL-3;

[0018] 3.2, implement the Fast-Speech2 model according to the Fast-Speech2 paper, train it on the dataset in 3.1 to get the pre-trained model;

[0019] 3.3, on the local server, input a piece of Chinese into the model to test the synthetic speech;

[0020] Step 4: Design of backend services for the video generation system.

[0021] 4.1, collect the Lip Reading Sentences 2 (LRS2) dataset;

[0022] 4.2, implement the Wav2Lip video generation network model according to the paper, train it on the dataset in 4.1 to get the pre-trained model;

[0023] 4.3, on the local server, use a piece of audio and a picture to input into the model of 4.2 to generate a video. In the above scheme, the video generation system is based on the speech synthesis system. It can use the speech generated by the speech synthesis system, or the speech uploaded by any user, and any face picture. The speech and picture are transmitted to the back-end server through the JSP page. The back-end loads the trained video generation model to complete the video generation, and then transmits the video to the front-end through the framework. It also adds the functions of adding, deleting, querying, batch deleting, downloading synthesized videos, and speed-changing synthesized videos, thus realizing a complete text-driven video generation system.

[0024] As a further optimization scheme of the present invention, multiple groups of user names and passwords can be added in step 1 to support multiple users to log in to the system at the same time and perform speech synthesis and video generation experience.

[0025] As a further optimization scheme of the present invention, in step 2, in order to make this project more distinctive, any picture that the user is interested in can be selected when displayed on the front end. In addition, in order to facilitate the user to further refine the generated voice and video, a synthesized voice download, a synthesized voice speed change module, a generated video download and a generated video speed change module are newly added to the front-end framework code.

[0026] As a further optimization scheme of the present invention, the pre-trained model of the Fast-Speech2 network trained on the AISHELL-3 Chinese data set is finally obtained in step 3, and the speech generated by the model is closer to the voice of the real speaker;

[0027] As a further optimization scheme of the present invention, the video generation pre-training model obtained by training the Wav2Lip network on the LipReading Sentences 2 (LRS2) dataset in step 4 is natural and realistic.

[0028] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0029] ① The quality of the generated speech is high: the present invention can utilize AISHELL-3, a large, high-quality, publicly available Chinese speech synthesis dataset, and train it on the latest speech synthesis network Fast-Speech2 network to obtain a pre-trained model. Subsequently, the input text generates audio that is extremely close to the voice of a real speaker;

[0030] ② The quality of the generated video is high: the present invention can use the latest video generation network Wav2Lip to generate high-quality videos by inputting a piece of audio and any picture of the speaker;

[0031] ③ The framework of front-end and back-end interaction is highly scalable: The front-end and back-end interaction interface provided by the present invention can be extended to any task that requires calling the back-end deep learning model. To prove its strong scalability, the system of the present invention also provides a voice authentication module, which can input any audio and use the back-end model to determine whether it is artificially synthesized;

[0032] ④ The system supports multiple users to log in at the same time, with high concurrency: the user login interface provided by the present invention can register multiple different users, and adopts a multi-threaded approach to support multiple users to use the system at the same time;

[0033] ⑤ Good user experience: The present invention takes into account that users may have their own customized requirements for the generated audio and video, such as adjusting the audio and video speech speed and downloading audio and video. In the design of this system, relevant module functions are added.

[0034] In general, the present invention solves the problem that there is no speech synthesis system and video generation system in the existing scenario introduced in the background technology. The method can use the public high-quality AISHELL-3 Chinese data set and Fast-Speech2 network to complete high-quality speech synthesis, and can also use the public Lip Reading Sentences 2 (LRS2) data set and Wav2Lip network to complete high-quality video generation. The JavaScript framework is combined to complete the interaction between the front end and the back end, and the speech synthesis system and the video generation system are integrated together to form a text-driven video generation system. Since the system integrates the download of synthesized speech, the speed change module of synthesized speech, the download of generated video and the speed change module of generated video, the user experience will be better. In terms of synthesized speech, the system integrates the Fast-Speech2 model with the best synthesized speech effect at present, and trains the model in Chinese speech synthesis. Compared with other existing technical solutions, the solution can synthesize high-quality Chinese speech. In terms of video generation, the system adopts the Wav2Lip model, and its video generation effect reaches a relatively advanced level in the industry. In addition, the system supports high concurrency and can support multiple people to access and execute generation tasks at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of an implementation method of a text-driven video generation system. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0037] Embodiment: A method for implementing a text-driven video generation system, the specific steps are as follows:

[0038] Step 1: User login module design,

[0039] 1.1, create a MySQL database to store usernames and passwords;

[0040] 1.2, insert the username and password into the database;

[0041] Step 2: Front-end design of text-driven video generation system.

[0042] 2.1, select any picture that the user is interested in as the core picture displayed on the front-end interface;

[0043] 2.2, Use JavaScript language to build the front-end interface and insert the above pictures into the front-end page;

[0044] 2.3. Based on 2.2, the front-end language is used to implement the adding, deleting, querying, batch deleting, synthesized speech downloading, synthesized speech speed change module of the speech synthesis system, as well as the adding, deleting, querying, batch deleting, generating video downloading, and generating video speed change module of the video generation system.

[0045] 2.4, the interaction with the backend is completed by using the http protocol combined with the JSON data format. Two interfaces are mainly designed: speech synthesis interface and video generation interface. The speech synthesis interface is any API containing the output file name and input text, such as any API address in the format of " / api / text?filename={filename}&text={text}", where filename and text represent the voice file name and text content respectively. The request method adopts Get, and the returned data format is designed in Json format, including the status code and the returned message. The video generation interface is any API containing the video file name, voice type, voice file name and picture file name, such as the format of " / api / video?mp4name={mp4name}&checkpointname={checkpointname}&audioname={audiona me}&picname={picname}", where mp4name, checkpointname, audioname and picname represent the video file name, voice type, voice file name and picture file name respectively. The request method adopts Get, and the returned data format is designed in JSON format, including the status code and the returned message;

[0046] Step 3: Design of backend service for speech synthesis system.

[0047] 3.1, collect the Chinese speech synthesis dataset AISHELL-3;

[0048] 3.2, implement the Fast-Speech2 model according to the Fast-Speech2 paper, train it on the dataset in 3.1 to get the pre-trained model;

[0049] 3.3, on the local server, input a piece of Chinese into the model to test the synthetic speech;

[0050] Step 4: Design of backend services for the video generation system.

[0051] 4.1, collect the Lip Reading Sentences 2 (LRS2) dataset;

[0052] 4.2, implement the Wav2Lip video generation network model according to the paper, train it on the dataset in 4.1 to get the pre-trained model;

[0053] 4.3, on the local server, use a piece of audio and a picture to input into the model in 4.2 to generate a video;

[0054] The schematic diagram of the implementation method of a text-driven video generation system proposed by the present invention is as follows: Figure 1 As shown in the figure. This figure mainly consists of three parts: user login and front-end design, speech synthesis system back-end service design, and video generation system back-end service design. The first part is to design the login module, store the administrator account password, and design the interface for interaction with the back-end. The second part is the development of the speech synthesis model that the front-end needs to call. The Chinese speech synthesis data set AISHELL-3 is collected, and the model training is completed on the Fast-Speech2 model to obtain a better speech synthesis model. The third part is the development of the video generation model that the front-end needs to call. Using the collected Lip Reading Sentences 2 (LRS2) data set, the video generation is obtained by training on the Wav2Lip model. The model can generate a more realistic video using a single picture and a piece of audio. At this point, the front-end can easily realize speech synthesis and video generation tasks by transparently transmitting parameters to the back-end and calling the developed model.

[0055] The above is an implementation method of this patent. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that he or she can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for implementing a text-driven video generation system, characterized in that the method comprises the following steps: Step 1: Design the user login module. Step 2: Design the front end of the text-driven video generation system Step 3: Design the backend service of the speech synthesis system Step 4: Design the backend service of the video generation system; Among them, step 1, design the user login module, as follows: 1.1, establish a MySQL database to store usernames and passwords; 1.2, insert the username and password into the database; Step 2: Design the front end of the text-driven video generation system, as follows: 2.1, select any picture that the user is interested in as the core picture displayed on the front-end interface; 2.2, Use JavaScript, HTML and CSS to build the front-end interface, and insert the above pictures into the front-end page; 2.3, based on 2.2, the front-end language is used to implement the addition, deletion, query, batch deletion, synthesized speech download, and synthesized speech speed change modules of the speech synthesis system, as well as the addition, deletion, query, batch deletion, generated video download, and generated video speed change modules of the video generation system. 2.4, the interaction with the backend is completed by using the http protocol combined with the JSON data format. Step 3: Design the backend service of the speech synthesis system, as follows: 3.1, collect the Chinese speech synthesis dataset AISHELL-3; 3.2, implement the Fast-Speech2 model based on Fast-Speech2, train it on the AISHELL-3 dataset in 3.1, and obtain a pre-trained model for Chinese speech synthesis; 3.3, on the local server, use a Chinese input to the pre-trained model of Chinese speech synthesis to test the synthesized speech, Step 4: Design of the backend service of the video generation system, as follows: 4.1, collect the Lip Reading Sentences 2 (LRS2) dataset; 4.2, implement the Wav2Lip video generation network model, train it on the Lip Reading Sentences 2 (LRS2) dataset in 4.1, and obtain a pre-trained model for video generation; 4.3, on the local server, use a piece of audio and picture to input the pre-trained model of video generation in 4.2 to generate video, Put the two different tasks of speech synthesis and video generation into one system, In step 2, any picture that the user is interested in is selected as the core picture displayed on the front-end interface, and different front-end and back-end interaction interfaces are customized for different tasks.

Images