Training method of video voice generation model, video synthesis method and related equipment
By building and training audio to audio and video to audio models, the problems of low voice quality and poor adaptability in video speech synthesis are solved, and higher quality and adaptable video speech generation are achieved.
Patent Information
- Application Number
- CN202410975092.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-19
AI Technical Summary
The existing video speech synthesis technology is not high in the quality of the generated speech and is difficult to adapt to the needs of different target scenarios, resulting in a reduced video speech synthesis effect.
The first model of audio to audio and the second model of video to audio are constructed, and the decoder of the second model is pre-trained and the second model is initialized, and the video data in the target scene is trained to generate high-quality video voice.
The quality of voice generated in video speech synthesis is improved, making the synthesis effect more suitable for the needs of different scenarios, and improving the overall effect of video speech synthesis.
Smart Images

Figure CN119028359A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of financial technology, and in particular to a training method for a video speech generation model, a video synthesis method and related equipment. Background Art
[0002] At present, there are some applications related to video speech synthesis in the field of financial customer service. Through video speech synthesis, video pixels can be combined with natural language text prompts to generate audio synchronized with the video content, thereby creating a highly flexible audio track for the video, including dialogue, sound effects, music, etc. However, the existing video speech synthesis still has the problem of low quality of generated speech and difficulty in adapting to the needs of different target scenarios, which reduces the effect of video speech synthesis. Summary of the invention
[0003] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a training method, a video synthesis method and related equipment for a video speech generation model that can be applied to financial technology or other related fields. Its main purpose is to improve the quality of generated speech in video speech synthesis, thereby improving the synthesis effect.
[0004] The technical solution of the present invention is as follows:
[0005] A first aspect of the present invention provides a method for training a video speech generation model, comprising:
[0006] Constructing a first audio-to-audio model and a second video-to-audio model, wherein the first model includes an audio encoder and a first audio decoder; the second model includes a video frame encoder, an identity encoder, and a second audio decoder, and the first audio decoder and the second audio decoder have the same structure;
[0007] Collecting a large amount of mono data to pre-train the first model, and saving parameters of the first audio decoder when the first model completes pre-training;
[0008] Initializing the second audio decoder according to the parameters of the first audio decoder;
[0009] The video data set collected in the target scene is input into the initialized second model for training, and the training is completed until the preset convergence condition is met to obtain a video speech generation model.
[0010] In one embodiment, collecting a large amount of mono data to pre-train the first model, and saving the parameters of the first audio decoder when the first model completes the pre-training, includes:
[0011] Collect a large amount of mono data,
[0012] Inputting the monophonic data into the first model, extracting speech features through the audio encoder, and reconstructing the speech features through the first audio decoder to generate a reconstructed audio signal;
[0013] Back-propagating and updating the weights of the first model according to the generated reconstructed audio signal until pre-training is completed;
[0014] Obtain and save parameters of the first audio decoder in the first model that has completed pre-training.
[0015] In one embodiment, the initializing the second audio decoder according to the parameters of the first audio decoder specifically includes:
[0016] The parameters of the first audio decoder are used as initial parameters of the second audio decoder to complete the initialization of the second audio decoder.
[0017] In one embodiment, a video data set collected in a target scene is input into an initialized second model for training until a preset convergence condition is met, and the training is completed to obtain a video speech generation model, including:
[0018] Acquire a video data set collected in a target scene, and extract video frames and original audio from the video data set;
[0019] Inputting the video frame into a second model, extracting visual features through the video encoder, and extracting speaking face features through the identity encoder;
[0020] The second audio decoder predicts and generates corresponding target audio according to the visual features and the speaker's facial features;
[0021] The weight of the second model is updated according to the target audio and the original audio under the supervision of the loss function until a preset convergence condition is met to obtain a video speech generation model.
[0022] In one embodiment, the signal form of the original audio includes an original waveform and a Mel spectrum.
[0023] A second aspect of the present invention provides a video synthesis method, comprising:
[0024] Get the silent video to be synthesized;
[0025] Inputting the silent video to be synthesized into a pre-trained video speech generation model to generate corresponding audio;
[0026] Synthesize the silent video and the generated audio to obtain a video with sound;
[0027] Wherein, the pre-trained video speech generation model is obtained by adopting the training method as described above.
[0028] A third aspect of the present invention provides a training device for a video speech generation model, comprising:
[0029] A construction module, configured to construct a first audio-to-audio model and a second video-to-audio model, wherein the first model includes an audio encoder and a first audio decoder; the second model includes a video frame encoder, an identity encoder, and a second audio decoder, and the first audio decoder and the second audio decoder have the same structure;
[0030] A first training module, used for collecting a large amount of monophonic data to pre-train the first model, and saving the parameters of the first audio decoder when the first model completes pre-training;
[0031] an initialization module, configured to initialize the second audio decoder according to the parameters of the first audio decoder;
[0032] The second training module is used to input the video data set collected in the target scene into the initialized second model for training, and the training is completed until the preset convergence condition is met to obtain a video speech generation model.
[0033] A fourth aspect of the present invention provides a video synthesis device, comprising:
[0034] An acquisition module, used for acquiring a silent video to be synthesized;
[0035] An audio generation module, used for inputting the silent video to be synthesized into a pre-trained video speech generation model to generate corresponding audio;
[0036] A synthesis module, used for synthesizing the silent video and the generated audio to obtain a video with sound;
[0037] Wherein, the pre-trained video speech generation model is obtained by adopting the training method as described above.
[0038] A fifth aspect of the present invention provides an electronic device, comprising at least one processor; and
[0039] a memory communicatively connected to the at least one processor; wherein,
[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the above-mentioned video speech generation model or execute the above-mentioned video synthesis method.
[0041] A sixth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned video speech generation model training method or execute the above-mentioned video synthesis method.
[0042] Beneficial effects: The present invention discloses a training method for a video speech generation model, a video synthesis method and related equipment. Compared with the prior art, the embodiment of the present invention constructs a first audio-to-audio model and a second video-to-audio model. The first audio decoder in the first model and the second audio decoder in the second model have the same structure; a large amount of mono data is collected to pre-train the first model, and the parameters of the first audio decoder are saved when the first model completes pre-training; the second audio decoder is initialized according to the parameters of the first audio decoder; the video data set collected in the target scene is input into the initialized second model for training until the preset convergence condition is met, and the training is completed to obtain a video speech generation model. The model is initialized by a pre-trained audio decoder, so that the model can adapt to the characteristics of the target scene data set while retaining the pre-trained speech features, thereby improving the quality of the generated speech in the video speech synthesis, thereby improving the synthesis effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0044] Figure 1 A flow chart of a method for training a video speech generation model provided by an embodiment of the present invention;
[0045] Figure 2 A flowchart of step S200 in the training method of the video speech generation model provided in an embodiment of the present invention;
[0046] Figure 3 A flowchart of step S400 in the method for training a video speech generation model provided by an embodiment of the present invention;
[0047] Figure 4 A flow chart of a video synthesis method provided by an embodiment of the present invention;
[0048] Figure 5 A schematic diagram of functional modules of a training device for a video speech generation model provided by an embodiment of the present invention;
[0049] Figure 6 A schematic diagram of functional modules of a video synthesis device provided by an embodiment of the present invention;
[0050] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.
[0052] At present, there are some applications related to video speech synthesis in the field of financial customer service. Through video speech synthesis, video pixels can be combined with natural language text prompts to generate audio synchronized with the video content, thereby creating a highly flexible audio track for the video, including dialogue, sound effects, music, etc. However, the existing video speech synthesis still has the problem of low quality of generated speech and difficulty in adapting to the needs of different target scenarios, which reduces the effect of video speech synthesis.
[0053] In order to solve the above problems, the present invention proposes a training method for a video speech generation model and a video synthesis method. The system architecture of the training method for the video speech generation model or the video synthesis method provided in this embodiment may include a first terminal device, a second terminal device, a third terminal device, a network and a server. The network is used to provide a medium for a communication link between the first terminal device, the second terminal device, the third terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.
[0054] The user may use the first terminal device, the second terminal device, and the third terminal device to interact with the server through the network to receive or send messages, etc. Various communication client applications may be installed on the first terminal device, the second terminal device, and the third terminal device, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).
[0055] The first terminal device, the second terminal device, and the third terminal device may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0056] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device, the second terminal device, and the third terminal device (for example only). The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.
[0057] It should be noted that the video speech generation model training method or video synthesis method provided in the embodiment of the present invention can generally be executed by the first terminal device, the second terminal device or the third terminal device. Correspondingly, the video speech generation model training device or video synthesis device provided in the embodiment of the present invention can also be set in the first terminal device, the second terminal device or the third terminal device.
[0058] Alternatively, the training method or video synthesis method of the video speech generation model provided by the embodiment of the present invention may also be generally performed by a server. Accordingly, the training device or video synthesis device of the video speech generation model provided by the embodiment of the present invention may generally be arranged in a server. The training method or video synthesis method of the video speech generation model provided by the embodiment of the present invention may also be performed by a server or server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server. Accordingly, the training device or video synthesis device of the video speech generation model provided by the embodiment of the present invention may also be arranged in a server or server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server.
[0059] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0060] See also Figure 1 , Figure 1 The flowchart of one embodiment of the training method of the video speech generation model provided by the present invention. Figure 1 As shown, the method specifically comprises the following steps:
[0061] S100, constructing a first audio-to-audio model and a second video-to-audio model, wherein the first model includes an audio encoder and a first audio decoder; the second model includes a video frame encoder, an identity encoder and a second audio decoder, and the first audio decoder and the second audio decoder have the same structure.
[0062] In this embodiment, a first audio-to-audio model (A2A model) and a second video-to-audio model (V2A model) are constructed respectively, where the A2A model generally refers to an audio-based artificial intelligence model that can be used for various tasks, such as speech recognition, speech synthesis, audio classification, etc.; the V2A model is an artificial intelligence model that generates synchronized audio corresponding to the device based on the visual features of the input video, thereby achieving the effect of adding an audio signal to a silent video.
[0063] The first model in this embodiment includes an audio encoder and a first audio decoder, wherein the structure of the audio encoder is opposite to that of the first audio decoder, that is, transposed convolution is used to replace convolution, or convolution is used to replace transposed convolution, etc. The specific structure of the audio encoder can adopt a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer network based on a self-attention mechanism, etc.; the second model includes a video frame encoder, an identity encoder, and a second audio decoder, and the structures of the first audio decoder and the second audio decoder are the same. In other words, the structure of the decoder of the second model is the same as that of the first model, while the structure of the encoder is different according to the input modality. By using a decoder with the same structure for audio decoding, the training and learning of the audio data by the first model can be retained by the second model to improve the quality and efficiency of its generated speech.
[0064] S200. Collect a large amount of mono data to pre-train the first model, and save the parameters of the first audio decoder when the first model completes pre-training.
[0065] In this embodiment, a large amount of mono data is first collected to pre-train the first model, and then a generative model from audio to audio is obtained. The purpose of pre-training is to allow the first audio decoder to learn general speech features without relying on a specific video data set. When the first model completes pre-training, the parameters of the first audio decoder are saved for parameter sharing with the second model, so that the second model can retain the learning results of the first model on speech features.
[0066] In one embodiment, Figure 2 As shown, the collecting of a large amount of mono data to pre-train the first model and saving the parameters of the first audio decoder when the first model completes the pre-training include:
[0067] S201, collect a large amount of mono data,
[0068] S202, inputting the monophonic data into a first model, extracting speech features through the audio encoder, and reconstructing the speech features through the first audio decoder to generate a reconstructed audio signal;
[0069] S203, back-propagating and updating the weights of the first model according to the generated reconstructed audio signal until the pre-training is completed;
[0070] S204: Obtain and save parameters of the first audio decoder in the first model that has completed pre-training.
[0071] In this embodiment, a large amount of mono data is collected for training the first model. The mono data can be from different sources, such as speech, music, environmental sound, etc. Mono audio is the most basic audio format, which contains only one channel, so all audio information is mixed in this single channel. Since the amount of mono data is small, it is relatively easy to process, and it can also reduce the complexity of the model to a certain extent and improve the training efficiency. Before training, the mono data can also be preprocessed, such as cleaning the data, removing noise and irrelevant parts; or standardizing the audio signal to make it have a uniform format and volume, etc., to improve the training effect.
[0072] After that, the mono data is input into the first model, and the input audio signal is encoded into a low-dimensional feature vector through the audio encoder to extract speech features, such as Mel-frequency cepstral coefficients (MFCC), Mel-frequency spectrum energy features (MEL), etc. The first audio decoder performs audio reconstruction based on the extracted features to generate a reconstructed audio signal, and the reconstructed audio signal output by forward propagation calculation based on the first model is updated by back propagation under the supervision of loss functions such as mean square error (MSE), cross entropy loss, etc. until the model converges and the pre-training is completed, for example, the number of trainings reaches a preset number or the value of the loss function is less than a preset value, etc. After the pre-training is completed, the weight and architecture of the first model are saved, and the parameters of the first decoder are extracted for subsequent training of the second model.
[0073] S300: Initialize the second audio decoder according to parameters of the first audio decoder.
[0074] In this embodiment, based on the pre-training results of the first model, the first audio decoder has learned general speech features without relying on a specific video data set, and the second audio decoder in the second video-to-audio model is initialized according to the parameters of the first audio decoder. Specifically, since the two audio decoders have the same structure, the parameters of the first audio decoder are directly used as the initial parameters of the second audio decoder, thereby completing the initialization of the second audio decoder, so that the second model can retain the pre-trained speech features during training, thereby improving the video speech generation effect.
[0075] S400: Input the video data set collected in the target scene into the initialized second model for training, and complete the training until the preset convergence condition is met to obtain a video speech generation model.
[0076] In this embodiment, the initialized second model is fine-tuned and trained using a video data set collected under a target scenario. The target scenario can be flexibly set according to demand, such as a financial service scenario, a feedback complaint scenario, an education and training scenario, or an interaction scenario involving dialects, etc. Different target scenarios may have different speech characteristics. Therefore, the second model is fine-tuned and trained using a video data set collected under the target scenario. The purpose of fine-tuning is to adapt the second video-to-speech model to a specific video data set while retaining the pre-trained speech features, so that the video speech generation can adapt to the needs of different scenarios and improve the speech synthesis effect.
[0077] In one embodiment, Figure 3 As shown, the video data set collected in the target scene is input into the initialized second model for training until the preset convergence condition is met, and the training is completed to obtain a video speech generation model, including:
[0078] S301, obtaining a video data set collected in a target scene, and extracting video frames and original audio in the video data set;
[0079] S302, inputting the video frame into a second model, extracting visual features through the video encoder, and extracting speaking face features through the identity encoder;
[0080] S303, predicting and generating corresponding target audio according to the visual features and the speaker's facial features through the second audio decoder;
[0081] S304. Update the weight of the second model according to the target audio and the original audio under the supervision of the loss function until a preset convergence condition is met to obtain a video speech generation model.
[0082] In this embodiment, in the fine-tuning training stage, a video data set collected in the target scene is obtained, and the video data set contains the sound types that need to be learned by the second model in the target scene. The video data set is preprocessed, including extracting video frames, audio signals, etc., and the video frames and original audio are extracted. The signal form of the original audio may include the original waveform and Mel spectrum. Similarly, the audio signal form of the first model pre-training may also include the original waveform and Mel spectrum, so as to realize the generation of video speech in different signal forms to meet different needs and scenarios.
[0083] Then, the video frame is input into the second model, and the visual features are extracted by the video encoder in the second model. For example, 3D convolution and ResNet-18 can be used to extract the visual features of the video frame, and the speaker face features can be extracted by the identity encoder. The identity encoder can use a pre-trained face or speaker encoder, such as an identity encoder based on a deep neural network, an identity encoder based on 3DMM, etc.; the extracted visual features and speaker face features are sent to the second audio decoder for predictive decoding to generate the target audio corresponding to the video frame. Specifically, for the two forms of audio signals, the target audio can be generated in different ways. For example, for the original waveform, a discriminator can be used based on a generative adversarial network to improve the authenticity of the generated waveform; for the spectrum, a Conformer-based method can be used to generate a waveform from the Mel spectrum using a pre-trained neural vocoder, so that the model can work on different target outputs. Whether it is the original waveform or the Mel spectrum, high-quality speech can be generated, which can not only meet different needs and scenarios, but also avoid the use of an additional vocoder to convert the output format.
[0084] During training, based on the target audio generated by the second model through forward propagation calculation, the weights of the second model are updated by back propagation under the supervision of loss functions such as mean square error (MSE) and cross entropy loss. The training is completed when the model converges. For example, the number of training times reaches a preset number or the value of the loss function is less than a preset value, etc. The second model after training can receive silent video input, extract low-dimensional features and generate corresponding speech output. The model is initialized through a pre-trained audio decoder, so that the model can adapt to the characteristics of the target scene data set while retaining the pre-trained speech features, thereby improving the performance of the video-to-speech model and the quality of the generated speech in video speech synthesis, thereby improving the synthesis effect.
[0085] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.
[0086] Another embodiment of the present invention provides a video synthesis method, such as Figure 4 As shown, the method comprises the following steps:
[0087] S500, obtaining a silent video to be synthesized;
[0088] S600, inputting the silent video to be synthesized into a pre-trained video speech generation model to generate corresponding audio;
[0089] S700: synthesizing the silent video and the generated audio to obtain a video with sound
[0090] Wherein, the pre-trained video speech generation model is obtained by adopting the training method as mentioned above.
[0091] In this embodiment, after the video speech generation model is trained by the training method as described above, end-to-end video synthesis can be achieved in a variety of application scenarios, such as financial service scenarios, feedback complaint scenarios, education and training scenarios, or interactive scenarios involving dialects, etc. The video speech generation model can generate corresponding audio according to the visual content of the video, and efficiently add detailed audio tracks to silent videos, including dialogues, sound effects, music, etc., to achieve the effect of audio-visual synchronization.
[0092] Specifically, a silent video to be synthesized is obtained, and the silent video to be synthesized is input into a pre-trained video speech generation model. The model can extract visual features of the video frame, such as lip shape features, identity features, background features, etc., and decode the extracted visual features to generate corresponding audio. The silent video and the generated audio are synthesized to obtain a sound video, thereby realizing high-quality video speech synthesis that meets the needs of different scenarios.
[0093] Another embodiment of the present invention provides a training device for a video speech generation model, such as Figure 5 As shown, the training device 1 comprises:
[0094] A construction module 11 is used to construct a first audio-to-audio model and a second video-to-audio model, wherein the first model includes an audio encoder and a first audio decoder; the second model includes a video frame encoder, an identity encoder, and a second audio decoder, and the first audio decoder and the second audio decoder have the same structure;
[0095] A first training module 12 is used to collect a large amount of monophonic data to pre-train the first model, and save the parameters of the first audio decoder when the first model completes pre-training;
[0096] An initialization module 13, configured to initialize the second audio decoder according to the parameters of the first audio decoder;
[0097] The second training module 14 is used to input the video data set collected in the target scene into the initialized second model for training, and the training is completed until the preset convergence condition is met to obtain a video speech generation model.
[0098] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the training execution process of the video speech generation model than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.
[0099] In one embodiment, the first training module 12 includes:
[0100] The acquisition unit is used to collect a large amount of mono data.
[0101] A first input unit, configured to input the monophonic data into a first model, extract speech features through the audio encoder, and reconstruct the speech features through the first audio decoder to generate a reconstructed audio signal;
[0102] A pre-training unit, configured to update the weights of the first model by back-propagation according to the generated reconstructed audio signal until the pre-training is completed;
[0103] A parameter saving unit is used to obtain and save the parameters of the first audio decoder in the first model that has completed pre-training.
[0104] In one embodiment, the initialization module 13 is specifically used to:
[0105] The parameters of the first audio decoder are used as initial parameters of the second audio decoder to complete the initialization of the second audio decoder.
[0106] In one embodiment, the second training module 14 includes:
[0107] An acquisition unit, used to acquire a video data set collected in a target scene, and extract video frames and original audio from the video data set;
[0108] A second input unit, configured to input the video frame into a second model, extract visual features through the video encoder, and extract speaker face features through the identity encoder;
[0109] A prediction generation unit, configured to predict and generate corresponding target audio according to the visual features and the speaker's facial features through the second audio decoder;
[0110] A training update unit is used to update the weight of the second model according to the target audio and the original audio under the supervision of a loss function until a preset convergence condition is met to obtain a video speech generation model.
[0111] In one embodiment, the signal form of the original audio includes an original waveform and a Mel spectrum.
[0112] Another embodiment of the present invention provides a video synthesis device, such as Figure 6 As shown, the device 2 comprises:
[0113] An acquisition module 21 is used to acquire a silent video to be synthesized;
[0114] The audio generation module 22 is used to input the silent video to be synthesized into a pre-trained video speech generation model to generate corresponding audio;
[0115] A synthesis module 23, used for synthesizing the silent video and the generated audio to obtain a video with sound;
[0116] Wherein, the pre-trained video speech generation model is obtained by adopting the training method as described above.
[0117] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which are more suitable for describing the video synthesis execution process than programs. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.
[0118] Another embodiment of the present invention provides an electronic device, such as Figure 7 As shown, the electronic device 10 includes:
[0119] One or more processors 110 and memory 120, Figure 7 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.
[0120] The processor 110 is used to complete various control logics of the electronic device 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.
[0121] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the training method of the video speech generation model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the electronic device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the training method of the video speech generation model or the video synthesis method in the above method embodiment.
[0122] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the electronic device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0123] One or more units are stored in the memory 120, and when executed by one or more processors 110, the training method of the video speech generation model in any of the above method embodiments is executed, for example, the following steps are executed:
[0124] Constructing a first audio-to-audio model and a second video-to-audio model, wherein the first model includes an audio encoder and a first audio decoder; the second model includes a video frame encoder, an identity encoder, and a second audio decoder, and the first audio decoder and the second audio decoder have the same structure;
[0125] Collecting a large amount of mono data to pre-train the first model, and saving parameters of the first audio decoder when the first model completes pre-training;
[0126] Initializing the second audio decoder according to the parameters of the first audio decoder;
[0127] The video data set collected in the target scene is input into the initialized second model for training, and the training is completed until the preset convergence condition is met to obtain a video speech generation model.
[0128] In one embodiment, collecting a large amount of mono data to pre-train the first model, and saving the parameters of the first audio decoder when the first model completes the pre-training, includes:
[0129] Collect a large amount of mono data,
[0130] Inputting the monophonic data into the first model, extracting speech features through the audio encoder, and reconstructing the speech features through the first audio decoder to generate a reconstructed audio signal;
[0131] Back-propagating and updating the weights of the first model according to the generated reconstructed audio signal until pre-training is completed;
[0132] Obtain and save parameters of the first audio decoder in the first model that has completed pre-training.
[0133] In one embodiment, the initializing the second audio decoder according to the parameters of the first audio decoder specifically includes:
[0134] The parameters of the first audio decoder are used as initial parameters of the second audio decoder to complete the initialization of the second audio decoder.
[0135] In one embodiment, a video data set collected in a target scene is input into an initialized second model for training until a preset convergence condition is met, and the training is completed to obtain a video speech generation model, including:
[0136] Acquire a video data set collected in a target scene, and extract video frames and original audio from the video data set;
[0137] Inputting the video frame into a second model, extracting visual features through the video encoder, and extracting speaking face features through the identity encoder;
[0138] The second audio decoder predicts and generates corresponding target audio according to the visual features and the speaker's facial features;
[0139] The weight of the second model is updated according to the target audio and the original audio under the supervision of the loss function until a preset convergence condition is met to obtain a video speech generation model.
[0140] In one embodiment, the signal form of the original audio includes an original waveform and a Mel spectrum.
[0141] Alternatively, the video synthesis method in any of the above method embodiments is executed, for example, the following steps are executed:
[0142] Get the silent video to be synthesized;
[0143] Inputting the silent video to be synthesized into a pre-trained video speech generation model to generate corresponding audio;
[0144] Synthesize the silent video and the generated audio to obtain a video with sound;
[0145] Wherein, the pre-trained video speech generation model is obtained by adopting the training method as described above.
[0146] The embodiment of the present invention further provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the training method of the video speech generation model in any of the above method embodiments is executed, for example, the following steps are performed:
[0147] Constructing a first audio-to-audio model and a second video-to-audio model, wherein the first model includes an audio encoder and a first audio decoder; the second model includes a video frame encoder, an identity encoder, and a second audio decoder, and the first audio decoder and the second audio decoder have the same structure;
[0148] Collecting a large amount of mono data to pre-train the first model, and saving parameters of the first audio decoder when the first model completes pre-training;
[0149] Initializing the second audio decoder according to the parameters of the first audio decoder;
[0150] The video data set collected in the target scene is input into the initialized second model for training, and the training is completed until the preset convergence condition is met to obtain a video speech generation model.
[0151] In one embodiment, collecting a large amount of mono data to pre-train the first model, and saving the parameters of the first audio decoder when the first model completes the pre-training, includes:
[0152] Collect a large amount of mono data,
[0153] Inputting the monophonic data into the first model, extracting speech features through the audio encoder, and reconstructing the speech features through the first audio decoder to generate a reconstructed audio signal;
[0154] Back-propagating and updating the weights of the first model according to the generated reconstructed audio signal until pre-training is completed;
[0155] Obtain and save parameters of the first audio decoder in the first model that has completed pre-training.
[0156] In one embodiment, the initializing the second audio decoder according to the parameters of the first audio decoder specifically includes:
[0157] The parameters of the first audio decoder are used as initial parameters of the second audio decoder to complete the initialization of the second audio decoder.
[0158] In one embodiment, a video data set collected in a target scene is input into an initialized second model for training until a preset convergence condition is met, and the training is completed to obtain a video speech generation model, including:
[0159] Acquire a video data set collected in a target scene, and extract video frames and original audio from the video data set;
[0160] Inputting the video frame into a second model, extracting visual features through the video encoder, and extracting speaking face features through the identity encoder;
[0161] The second audio decoder predicts and generates corresponding target audio according to the visual features and the speaker's facial features;
[0162] The weight of the second model is updated according to the target audio and the original audio under the supervision of the loss function until a preset convergence condition is met to obtain a video speech generation model.
[0163] In one embodiment, the signal form of the original audio includes an original waveform and a Mel spectrum.
[0164] Alternatively, the video synthesis method in any of the above method embodiments is executed, for example, the following steps are executed:
[0165] Get the silent video to be synthesized;
[0166] Inputting the silent video to be synthesized into a pre-trained video speech generation model to generate corresponding audio;
[0167] Synthesize the silent video and the generated audio to obtain a video with sound;
[0168] Wherein, the pre-trained video speech generation model is obtained by adopting the training method as described above.
[0169] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.
[0170] In summary, the training method, video synthesis method and related equipment of the video speech generation model disclosed in the present invention include: constructing a first audio-to-audio model and a second video-to-audio model, wherein the first audio decoder in the first model and the second audio decoder in the second model have the same structure; collecting a large amount of mono data to pre-train the first model, and saving the parameters of the first audio decoder when the first model completes pre-training; initializing the second audio decoder according to the parameters of the first audio decoder; inputting the video data set collected in the target scene into the initialized second model for training until the preset convergence condition is met, and the video speech generation model is obtained. The model is initialized by the pre-trained audio decoder, so that the model can adapt to the characteristics of the target scene data set while retaining the pre-trained speech features, thereby improving the quality of the generated speech in the video speech synthesis and thus improving the synthesis effect.
[0171] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.
[0172] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A training method for a video speech generation model, It is characterized in that include: Constructing a first audio-to-audio model and a second video-to-audio model, wherein the first model includes an audio encoder and a first audio decoder; the second model includes a video frame encoder, an identity encoder, and a second audio decoder, and the first audio decoder and the second audio decoder have the same structure; Collecting a large amount of mono data to pre-train the first model, and saving parameters of the first audio decoder when the first model completes pre-training; Initializing the second audio decoder according to the parameters of the first audio decoder; The video data set collected in the target scene is input into the initialized second model for training, and the training is completed until the preset convergence condition is met to obtain a video speech generation model.
2. The training method of the video speech generation model according to claim 1, It is characterized in that The collecting a large amount of monophonic data to pre-train the first model, and saving the parameters of the first audio decoder when the first model completes the pre-training, includes: Collect a large amount of mono data, Inputting the monophonic data into the first model, extracting speech features through the audio encoder, and reconstructing the speech features through the first audio decoder to generate a reconstructed audio signal; Back-propagating and updating the weights of the first model according to the generated reconstructed audio signal until pre-training is completed; Obtain and save parameters of the first audio decoder in the first model that has completed pre-training.
3. The training method of the video speech generation model according to claim 1, It is characterized in that Initializing the second audio decoder according to the parameters of the first audio decoder specifically includes: The parameters of the first audio decoder are used as initial parameters of the second audio decoder to complete the initialization of the second audio decoder.
4. The training method of the video speech generation model according to claim 1, It is characterized in that The video data set collected in the target scene is input into the initialized second model for training until the preset convergence condition is met, and the training is completed to obtain the video speech generation model, including: Acquire a video data set collected in a target scene, and extract video frames and original audio from the video data set; Inputting the video frame into a second model, extracting visual features through the video encoder, and extracting speaking face features through the identity encoder; Generate corresponding target audio by predicting according to the visual features and the speaker's facial features through the second audio decoder; The weight of the second model is updated according to the target audio and the original audio under the supervision of the loss function until a preset convergence condition is met to obtain a video speech generation model.
5. The training method of the video speech generation model according to claim 4, It is characterized in that The signal form of the original audio includes an original waveform and a Mel spectrum.
6. A video synthesis method, It is characterized in that include: Get the silent video to be synthesized; Inputting the silent video to be synthesized into a pre-trained video speech generation model to generate corresponding audio; Synthesize the silent video and the generated audio to obtain a video with sound; Wherein, the pre-trained video speech generation model is obtained by using the training method described in any one of claims 1-5.
7. A training device for a video speech generation model, It is characterized in that include: A construction module, configured to construct a first audio-to-audio model and a second video-to-audio model, wherein the first model includes an audio encoder and a first audio decoder; the second model includes a video frame encoder, an identity encoder, and a second audio decoder, and the first audio decoder and the second audio decoder have the same structure; A first training module, used for collecting a large amount of monophonic data to pre-train the first model, and saving the parameters of the first audio decoder when the first model completes pre-training; an initialization module, configured to initialize the second audio decoder according to the parameters of the first audio decoder; The second training module is used to input the video data set collected in the target scene into the initialized second model for training, and the training is completed until the preset convergence condition is met to obtain a video speech generation model.
8. A video synthesis device, It is characterized in that include: An acquisition module, used for acquiring a silent video to be synthesized; An audio generation module, used for inputting the silent video to be synthesized into a pre-trained video speech generation model to generate corresponding audio; A synthesis module, used for synthesizing the silent video and the generated audio to obtain a video with sound; Wherein, the pre-trained video speech generation model is obtained by using the training method described in any one of claims 1-5.
9. An electronic device, It is characterized in that comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the video speech generation model described in any one of claims 1-5, or execute the video synthesis method described in claim 6.
10. A non-volatile computer-readable storage medium, It is characterized in that The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the training method for the video speech generation model described in any one of claims 1-5, or execute the video synthesis method described in claim 6.
Citation Information
Patent Citations
Video synthesis model training method and device, video synthesis method and device, storage medium, program product and electronic equipment
CN113469292A
End-to-end (text-to-video) video synthesis method, system medium and application
CN114338959A
Method and device for generating speaking portrait video and training face rendering model
CN114581980A
Voice conversion model training method, voice conversion method and device
CN115273831A
Synthesizing method, device and equipment for talking video of virtual image and storage medium
CN115442543A
