Speech generation method based on deep learning

Abstract

The invention relates to the technical field of voice cloning, and particularly provides a voice generation method based on deep learning. The method comprises the steps of making a data set; building a text semantic model, and training the text semantic model by using the data set as training data to obtain a trained text semantic model; building a voice reasoning model, inputting the trained text semantic model into an audio generator in the voice reasoning model for processing, outputting an audio signal, and transmitting the audio signal to a comprehensive discriminator for judgment and scoring; and deploying the trained text semantic model and the voice reasoning model in a voice generation system for voice generation, the method breaks through the limitation of a traditional dubbing mode, improves the efficiency and flexibility of news program production, and ensures the timeliness and accuracy of news transmission.

Description

Technical Field

[0001] The present invention relates to the technical field of voice cloning, and in particular to a voice generation method based on deep learning. Background Art

[0002] In the field of radio and television media, the production requirements of news programs are extremely high. Among them, when editors are cutting and editing the materials, specific hosts are required to perform manual dubbing. However, this traditional dubbing model has many problems. First, editors need to make appointments in advance and coordinate the time of the hosts. Secondly, during the peak period of news program production, the number of materials that need dubbing increases sharply, resulting in the phenomenon of queuing for dubbing. Finally, the particularity of the news industry determines that emergencies may occur at any time. Due to the influx of the latest news, the content that has been dubbed needs to be modified. At this time, it is very difficult to quickly summon the host to re-dub. This series of problems may even lead to broadcast accidents, which in turn have a negative impact on the credibility of the TV station and the audience's viewing experience.

[0003] At present, the disclosed invention patent CN114863946A can realize timbre cloning with only a small amount of target speaker voice data, but in the synthesis stage, especially when the voice data is too single, it lacks a reference standard. The model does not have corresponding reference audio and reference text, it is difficult to adapt to the context and intention, the sound quality control is difficult, it is easily affected by noise, lacks coherence and consistency, and the generated speech lacks realism. In addition, since the method synthesizes audio in a one-time synthesis, the second half of the corresponding audio generated according to a large amount of target text is prone to distortion; the invention patent CN112233646A needs to replace Chinese characters with pinyin, convert Arabic numerals into Chinese characters, and convert time symbols into Chinese characters in the text processing stage. This method increases the risk of information loss, increases processing complexity, reduces processing efficiency, and lacks language flexibility. Therefore, it will lead to inaccurate information transmission, affect the user experience and system performance, and thus reduce the effectiveness and acceptance of the method in practical applications; the invention patent CN113436607A uses LSTM neural network, which requires a large data set, takes a long time to train, and is sensitive to noise and inconsistency of input data, resulting in a decrease in the quality of the generated audio. In addition, this method uses Relu delinearization, and during the training process, some neurons have a probability of continuously outputting zero values, which makes it impossible to update the weights, resulting in a decrease in the network's expressiveness. Summary of the invention

[0004] In view of this, the present invention provides a speech generation method based on deep learning, which is used to break the limitations of the traditional dubbing mode, improve the efficiency and flexibility of news program production, and ensure the timeliness and accuracy of news dissemination.

[0005] In a first aspect, the present invention provides a speech generation method based on deep learning, the method comprising:

[0006] Step 1: Create a data set, collect multiple audio files and manuscript files of different manuscripts corresponding to N hosts, pre-process the audio files, and separate the labels from the audio files, where N is a positive integer;

[0007] Step 2: Build a text semantic model, use the data set as training data, train the text semantic model, and obtain a trained text semantic model;

[0008] Step 3: Build a speech inference model, input the trained text semantic model into the audio generator in the speech inference model for processing, output the audio signal, and pass the audio signal to the comprehensive discriminator for judgment and scoring;

[0009] Step 4: Deploy the trained text semantic model and speech inference model in the speech generation system for speech generation.

[0010] Optionally, step 1 includes:

[0011] Step S11, collecting audio files and manuscript files of N presenters, and preprocessing them in a multi-threaded concurrent manner to obtain processed audio files, wherein N is a positive integer; the preprocessing method is: firstly, cutting the audio files according to the volume threshold, and taking the audio files below the volume threshold as the cutting points; and reducing the noise of the cut audio files by using an acoustic noise suppression method;

[0012] Step S12: Obtaining a label and an audio file by automatic speech recognition for the processed audio file;

[0013] Step S13: Check and correct the tags and audio files, and separate the tags from the audio files.

[0014] Optionally, the step S11 of cutting according to the volume threshold includes:

[0015] First, load the audio file and extract the signal, divide the signal into frames, calculate the root mean square volume of each frame, compare the volume of each frame with the threshold, record the number of frames where silence starts and ends, and start cutting if the consecutive silent frames exceed the shortest cutting interval; then cut the audio into multiple segments according to the detected non-silent segments, and save these segments as new audio files; reduce the noise of the new audio file through the acoustic noise suppression method; first, perform feature extraction, perform noise estimation in the signal feature space, then configure the suppression strategy, reconstruct the signal, and finally output clear audio.

[0016] Optionally, the automatic speech recognition method is adopted in step S12, including:

[0017] The audio signal is transformed into a frequency domain representation through short-time Fourier transform, and the audio features similar to the auditory characteristics of the human ear are extracted through the Mel-frequency cepstral coefficients; the audio features are then analyzed through the recurrent neural network RNN ​​and the corresponding text labels are output; the text output by the model is then post-processed to remove redundant spaces and punctuation marks, and saved, and the final result is output.

[0018] Optionally, the step 2 includes:

[0019] Step S21, configure the text semantic decoder, set the training hyperparameters, and convert the input text features into semantic representations; wherein, the process of configuring the text semantic decoder is: first, use the embedding layer to convert the phoneme ID and the semantic ID into a high-dimensional vector to capture their potential semantic relationship; then, use the gated recurrent unit GRU to construct the encoding layer to extract the context information of the input audio features; use the Transformer structure for parallel processing; the decoding layer gradually generates output semantics through the GRU and the self-attention mechanism; in the output layer, the hidden layer state is mapped to the target semantic space through the fully connected layer, and the Softmax is applied to generate the probability distribution of various semantic IDs; the entire text semantic model is trained and optimized using the cross entropy loss to achieve efficient text semantic conversion;

[0020] Step S22: build a feedforward network model, and process batch size, phoneme identifier and length, target semantic identifier and length features through forward propagation;

[0021] Step S23, calculate the loss and perform back propagation, perform learning rate warm-up and decay, obtain learning rate scheduler information, and update the text semantic model parameters through the ScaledAdam optimizer;

[0022] Step S24: When the text semantic model is initialized, check whether there are pre-trained weights; if there are pre-trained weights, load them to improve initialization performance;

[0023] Step S25: output the training log, save the latest checkpoint in the training process, and after the training is completed, save the final model weights to obtain the trained text semantic model.

[0024] Optionally, step 3 includes:

[0025] Step S31, constructing a speech inference model through an audio generator and a comprehensive discriminator; wherein a text encoder is constructed through a Transformer encoder and a self-attention mechanism, and a basic generator, a posterior encoder, and a spectrogram style encoder are constructed using a convolutional neural network CNN; a residual coupling block and a residual vector quantizer are constructed using a residual neural network, and finally the above modules are connected in series to form an audio generator; and then a comprehensive discriminator is constructed using CNN;

[0026] Step S32, input the audio data, audio features, and text information into the audio generator, use the text encoder to convert the input text information into a high-dimensional vector representation, process the input text through Transformer and self-attention mechanism, and extract the semantic information and contextual relationship in the text; process the audio features and audio data through the basic generator, map the input high-dimensional feature vector to an audio waveform, and generate an audio signal; use the posterior encoder to obtain the posterior features of the audio signal and estimate the authenticity of the generated audio signal; the residual coupling block uses the residual connection to avoid the gradient vanishing problem, and separates and combines according to the input potential representation; the spectrogram style encoder extracts the spectrogram style information, controls the style and emotional color of the generated audio, and uses CNN to extract deep features so that the model generates diversified audio styles; quantizes the features through the residual vector quantizer, and converts the continuous potential features into discrete representations to reduce the computational complexity;

[0027] Step S33, build a comprehensive discriminator through a basic discriminator and multiple sub-discriminators of different periods to evaluate the quality of the generated audio; the basic discriminator adopts a deep CNN architecture to process the original input audio signal, is responsible for extracting features from the input signal and making authenticity judgments, and evaluates the input real audio signal to obtain an authenticity score; the input audio is discriminated for multiple periods through a period discriminator, and the audio signal is independently discriminated for different periods to capture features in different frequency ranges; the input audio is transmitted to each period discriminator, processed through a custom convolutional architecture, and the input signal is subjected to a sliding window operation on different time scales to capture subtle changes and patterns, and finally the authenticity score of the corresponding period is returned.

[0028] Optionally, the voice system in step 4 includes: a host selection module, a segmentation mode module, a speech speed adjustment module, a dubbing content module, a dubbing generation module and a download module;

[0029] The host selection module is used to select a specific host, and the background will automatically match the reference audio and reference text of the selected host;

[0030] The segmentation mode module is used to provide multiple segmentation options, including segmentation when four sentences are recognized, segmentation when a period is recognized, and segmentation when the number of words counted reaches 45;

[0031] The speech rate adjustment module is used to adjust the speed of the synthesized audio, and its speed range is 0.4 to 2 times;

[0032] The dubbing content module is used to identify the dubbing content, including Chinese characters, punctuation marks, Arabic numerals and time symbols;

[0033] Generate dubbing module, which is used to divide the dubbing content into multiple short sentences according to the preset segmentation method and speech speed, and then use the text semantic model and speech reasoning model to gradually generate the audio corresponding to each short sentence, and finally integrate the audio corresponding to all short sentences to complete the splicing;

[0034] The download module is used to download the generated dubbing file to the local computer.

[0035] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the deep learning-based speech generation method in the first aspect or any possible implementation of the first aspect.

[0036] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to perform the deep learning-based speech generation method in the first aspect or any possible implementation of the first aspect.

[0037] In the technical solution provided by the present invention, the method includes preparing a data set, collecting multiple audio files and manuscript files of different manuscripts corresponding to N hosts, preprocessing the audio files, and separating labels from the audio files; building a text semantic model, using the data set as training data, training the text semantic model, and obtaining a trained text semantic model; building a speech inference model, inputting the trained text semantic model into an audio generator in the speech inference model for processing, outputting an audio signal, and passing the audio signal to a comprehensive discriminator for judgment and scoring; deploying the trained text semantic model and speech inference model in a speech generation system to perform speech generation. The method breaks the limitations of the traditional dubbing mode, improves the efficiency and flexibility of news program production, and ensures the timeliness and accuracy of news dissemination. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a deep learning-based speech generation method provided by an embodiment of the present invention;

[0040] Figure 2 A flowchart of another speech generation method based on deep learning provided by an embodiment of the present invention;

[0041] Figure 3 A framework diagram of a text semantic model provided by an embodiment of the present invention;

[0042] Figure 4 A framework diagram of a text semantic decoder in a text semantic model provided by an embodiment of the present invention;

[0043] Figure 5 A framework diagram of a speech inference model provided by an embodiment of the present invention;

[0044] Figure 6 A framework diagram of an audio generator in a speech inference model provided in an embodiment of the present invention;

[0045] Figure 7 A framework diagram of a comprehensive discriminator in a speech inference model provided by an embodiment of the present invention;

[0046] Figure 8 A schematic diagram of a speech generation system provided by an embodiment of the present invention;

[0047] Fig. 9 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0051] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0052] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0053] The present invention provides a speech generation method based on deep learning, such as Figure 1 and Figure 2 As shown, the method includes:

[0054] Step 1: Create a data set, collect multiple audio files and manuscript files of different manuscripts corresponding to N hosts, pre-process the audio files, and separate the labels from the audio files, where N is a positive integer.

[0055] In the embodiment of the present invention, step 1 includes:

[0056] Step S11, collecting audio files and manuscript files of N presenters, and preprocessing them in a multi-threaded concurrent manner to obtain processed audio files, wherein N is a positive integer; the preprocessing method is: firstly, cutting the audio files according to the volume threshold, and taking the audio files below the volume threshold as the cutting points; and reducing the noise of the cut audio files by using an acoustic noise suppression method;

[0057] In the embodiment of the present invention, the audio files and manuscript files collected in step S11 are original files. The audio is recorded using Dayang software and trimmed using a non-editing tool. The manuscript system is used to view the concatenation list, select the corresponding manuscript, delete the introduction, synchronous sound, etc., retain the main text content, and print the manuscript.

[0058] Step S12: Obtaining a label and an audio file by automatic speech recognition for the processed audio file;

[0059] Step S13: Check and correct the tags and audio files, and separate the tags from the audio files.

[0060] In the embodiment of the present invention, the step S11 includes:

[0061] First, load the audio file and extract the signal, divide the signal into frames, calculate the root mean square volume of each frame, compare the volume of each frame with the threshold, record the number of frames where silence starts and ends, and start cutting if the consecutive silent frames exceed the shortest cutting interval; then cut the audio into multiple segments according to the detected non-silent segments, and save these segments as new audio files; reduce the noise of the new audio file through the acoustic noise suppression method; first, perform feature extraction, perform noise estimation in the signal feature space, then configure the suppression strategy, reconstruct the signal, and finally output clear audio.

[0062] In the embodiment of the present invention, the automatic speech recognition method is adopted in step S12, including:

[0063] The audio signal is transformed into a frequency domain representation through short-time Fourier transform, and the audio features similar to the auditory characteristics of the human ear are extracted through the Mel-frequency cepstral coefficients; the audio features are then analyzed through the recurrent neural network RNN ​​and the corresponding text labels are output; the text output by the model is then post-processed to remove redundant spaces and punctuation marks, and saved, and the final result is output.

[0064] In the embodiment of the present invention, in step S13, it is necessary to detect spelling errors, typos, and missing characters in the text. According to the audit results, necessary modifications need to be made in the text. This involves rewriting the text, deleting erroneous parts, adding missing content, etc. A secondary audit is then performed to ensure the accuracy of the data.

[0065] Step 2: Build a text semantic model, use the data set as training data, train the text semantic model, and obtain a trained text semantic model.

[0066] In the embodiment of the present invention, Figure 3 and Figure 4 As shown, step 2 includes:

[0067] Step S21, configure the text semantic decoder, set the training hyperparameters, and convert the input text features into semantic representations; wherein, the process of configuring the text semantic decoder is: first, use the embedding layer to convert the phoneme ID and the semantic ID into a high-dimensional vector to capture their potential semantic relationship; then, use the gated recurrent unit GRU to construct the encoding layer to extract the context information of the input audio features; use the Transformer structure for parallel processing; the decoding layer gradually generates output semantics through the GRU and the self-attention mechanism; in the output layer, the hidden layer state is mapped to the target semantic space through the fully connected layer, and the Softmax is applied to generate the probability distribution of various semantic IDs; the entire text semantic model is trained and optimized using the cross entropy loss to achieve efficient text semantic conversion;

[0068] Step S22: build a feedforward network model, and process batch size, phoneme identifier and length, target semantic identifier and length features through forward propagation;

[0069] Step S23, calculate the loss and perform back propagation, perform learning rate warm-up and decay, obtain learning rate scheduler information, and update the text semantic model parameters through the ScaledAdam optimizer;

[0070] Step S24: When the text semantic model is initialized, check whether there are pre-trained weights; if there are pre-trained weights, load them to improve initialization performance;

[0071] Step S25: output the training log, save the latest checkpoint in the training process, and after the training is completed, save the final model weights to obtain the trained text semantic model.

[0072] Step 3: Build a speech inference model, input the trained text semantic model into the audio generator in the speech inference model for processing, output the audio signal, and pass the audio signal to the comprehensive discriminator for judgment and scoring.

[0073] In the embodiment of the present invention, Figures 5 to 7 As shown, step 3 includes:

[0074] Step S31, constructing a speech inference model through an audio generator and a comprehensive discriminator; wherein a text encoder is constructed through a Transformer encoder and a self-attention mechanism, and a basic generator, a posterior encoder, and a spectrogram style encoder are constructed using a convolutional neural network CNN; a residual coupling block and a residual vector quantizer are constructed using a residual neural network, and finally the above modules are connected in series to form an audio generator; and then a comprehensive discriminator is constructed using CNN;

[0075] Step S32, input the audio data, audio features, and text information into the audio generator, use the text encoder to convert the input text information into a high-dimensional vector representation, process the input text through Transformer and self-attention mechanism, and extract the semantic information and contextual relationship in the text; process the audio features and audio data through the basic generator, map the input high-dimensional feature vector to an audio waveform, and generate an audio signal; use the posterior encoder to obtain the posterior features of the audio signal and estimate the authenticity of the generated audio signal; the residual coupling block uses the residual connection to avoid the gradient vanishing problem, and separates and combines according to the input potential representation; the spectrogram style encoder extracts the spectrogram style information, controls the style and emotional color of the generated audio, and uses CNN to extract deep features so that the model generates diversified audio styles; quantizes the features through the residual vector quantizer, and converts the continuous potential features into discrete representations to reduce the computational complexity;

[0076] Step S33, build a comprehensive discriminator through a basic discriminator and multiple sub-discriminators of different periods to evaluate the quality of the generated audio; the basic discriminator adopts a deep CNN architecture to process the original input audio signal, is responsible for extracting features from the input signal and making authenticity judgments, and evaluates the input real audio signal to obtain an authenticity score; the input audio is discriminated for multiple periods through a period discriminator, and the audio signal is independently discriminated for different periods to capture features in different frequency ranges; the input audio is transmitted to each period discriminator, processed through a custom convolutional architecture, and the input signal is subjected to a sliding window operation on different time scales to capture subtle changes and patterns, and finally the authenticity score of the corresponding period is returned.

[0077] Step 4: Deploy the trained text semantic model and speech inference model in the speech generation system for speech generation.

[0078] In the embodiment of the present invention, Figure 8 As shown, the voice system in step 4 includes: a host selection module, a segmentation method module, a speech speed adjustment module, a dubbing content module, a dubbing generation module and a download module;

[0079] The host selection module is used to select a specific host, and the background will automatically match the reference audio and reference text of the selected host;

[0080] The segmentation mode module is used to provide multiple segmentation options, including segmentation when four sentences are recognized, segmentation when a period is recognized, and segmentation when the number of words counted reaches 45;

[0081] The speech rate adjustment module is used to adjust the speed of the synthesized audio, and its speed range is 0.4 to 2 times;

[0082] The dubbing content module is used to identify the dubbing content, including Chinese characters, punctuation marks, Arabic numerals and time symbols;

[0083] Generate dubbing module, which is used to divide the dubbing content into multiple short sentences according to the preset segmentation method and speech speed, and then use the text semantic model and speech reasoning model to gradually generate the audio corresponding to each short sentence, and finally integrate the audio corresponding to all short sentences to complete the splicing;

[0084] The download module is used to download the generated dubbing file to the local computer.

[0085] In addition, the system enables multi-threaded concurrent execution in the dubbing generation module, which supports multiple users to use it at the same time without interfering with each other. At the same time, the system introduces a cache cleanup mechanism. Specifically, first define and initialize a global variable, use the variable in the speech inference model to receive the intermediate value, and finally clear the variable in the host selection module, which effectively solves the problem of audio jamming after model switching.

[0086] In the embodiment of the present invention, N is set to 6, and multiple audio and manuscript files of different manuscripts of six hosts are first collected as raw data, preprocessed through multiple threads, audio is cut and noise is reduced according to the volume threshold, automatic speech recognition is used to obtain labels and check and correct errors, and labels and audio files are separated to produce a data set; then a text semantic model is built, a decoder is configured, hyperparameters are set, text features are converted into semantic representations, a feedforward network is built to process multiple features, loss backpropagation is calculated, the learning rate is adjusted and parameters are updated using a specific optimizer, pre-trained weights are initialized, checked and loaded, logs are recorded and checkpoints and final weights are saved; then a speech inference model is built, which consists of an audio generator containing multiple structures and a comprehensive discriminator, the former can convert input information into audio, and the latter evaluates audio quality; finally, the trained model is deployed in the speech generation system.

[0087] Each step of the embodiment of the present invention may be performed by an electronic device, which includes but is not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0088] In the technical solution provided by the present invention, the method includes preparing a data set, collecting multiple audio files and manuscript files of different manuscripts corresponding to N hosts, preprocessing the audio files, and separating labels from the audio files; building a text semantic model, using the data set as training data, training the text semantic model, and obtaining a trained text semantic model; building a speech inference model, inputting the trained text semantic model into an audio generator in the speech inference model for processing, outputting an audio signal, and passing the audio signal to a comprehensive discriminator for judgment and scoring; deploying the trained text semantic model and speech inference model in a speech generation system to perform speech generation. The method breaks the limitations of the traditional dubbing mode, improves the efficiency and flexibility of news program production, and ensures the timeliness and accuracy of news dissemination.

[0089] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned embodiment of the deep learning-based speech generation method.

[0090] Fig. 9 A schematic diagram of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the deep learning-based speech generation method in the embodiment is implemented. To avoid repetition, they are not described one by one here.

[0091] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will appreciate that Fig. 9 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0092] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0093] The memory 212 may be an internal storage unit of the electronic device 21, such as a hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 21. Further, the memory 212 may also include both an internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 may also be used to temporarily store data that has been output or is to be output.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A speech generation method based on deep learning, characterized in that: The method comprises: Step 1: Create a data set, collect multiple audio files and manuscript files of different manuscripts corresponding to N hosts, pre-process the audio files, and separate the labels from the audio files, where N is a positive integer; Step 2: Build a text semantic model, use the data set as training data, train the text semantic model, and obtain a trained text semantic model; Step 3: Build a speech inference model, input the trained text semantic model into the audio generator in the speech inference model for processing, output the audio signal, and pass the audio signal to the comprehensive discriminator for judgment and scoring; Step 4: Deploy the trained text semantic model and speech inference model in the speech generation system for speech generation.

2. The method according to claim 1, characterized in that The step 1 comprises: Step S11, collecting audio files and manuscript files of N presenters, and preprocessing them in a multi-threaded concurrent manner to obtain processed audio files, wherein N is a positive integer; the preprocessing method is: firstly, cutting the audio files according to the volume threshold, and taking the audio files below the volume threshold as the cutting points; and reducing the noise of the cut audio files by using an acoustic noise suppression method; Step S12: Obtaining a label and an audio file by automatic speech recognition for the processed audio file; Step S13: Check and correct the tags and audio files, and separate the tags from the audio files.

3. The method according to claim 2, characterized in that The step S11 of cutting according to the volume threshold includes: First, load the audio file and extract the signal, divide the signal into frames, calculate the root mean square volume of each frame, compare the volume of each frame with the threshold, record the number of frames where silence starts and ends, and start cutting if the consecutive silent frames exceed the shortest cutting interval; then cut the audio into multiple segments according to the detected non-silent segments, and save these segments as new audio files; reduce the noise of the new audio file through the acoustic noise suppression method; first, perform feature extraction, perform noise estimation in the signal feature space, then configure the suppression strategy, reconstruct the signal, and finally output clear audio.

4. The method according to claim 2, characterized in that: The automatic speech recognition method used in step S12 includes: The audio signal is transformed into a frequency domain representation through short-time Fourier transform, and the audio features similar to the auditory characteristics of the human ear are extracted through the Mel-frequency cepstral coefficients; the audio features are then analyzed through the recurrent neural network RNN ​​and the corresponding text labels are output; the text output by the model is then post-processed to remove redundant spaces and punctuation marks, and saved, and the final result is output.

5. The method according to claim 1, characterized in that The step 2 comprises: Step S21, configure the text semantic decoder, set the training hyperparameters, and convert the input text features into semantic representations; wherein, the process of configuring the text semantic decoder is: first, use the embedding layer to convert the phoneme ID and the semantic ID into a high-dimensional vector to capture their potential semantic relationship; then, use the gated recurrent unit GRU to construct the encoding layer to extract the context information of the input audio features; use the Transformer structure for parallel processing; the decoding layer gradually generates output semantics through the GRU and the self-attention mechanism; in the output layer, the hidden layer state is mapped to the target semantic space through the fully connected layer, and the Softmax is applied to generate the probability distribution of various semantic IDs; the entire text semantic model is trained and optimized using the cross entropy loss to achieve efficient text semantic conversion; Step S22: build a feedforward network model, and process batch size, phoneme identifier and length, target semantic identifier and length features through forward propagation; Step S23, calculate the loss and perform back propagation, perform learning rate warm-up and decay, obtain learning rate scheduler information, and update the text semantic model parameters through the ScaledAdam optimizer; Step S24: When the text semantic model is initialized, check whether there are pre-trained weights; if there are pre-trained weights, load them to improve initialization performance; Step S25: output the training log, save the latest checkpoint in the training process, and after the training is completed, save the final model weights to obtain the trained text semantic model.

6. The method according to claim 1, characterized in that The step 3 comprises: Step S31, constructing a speech inference model through an audio generator and a comprehensive discriminator; wherein a text encoder is constructed through a Transformer encoder and a self-attention mechanism, and a basic generator, a posterior encoder, and a spectrogram style encoder are constructed using a convolutional neural network CNN; a residual coupling block and a residual vector quantizer are constructed using a residual neural network, and finally the above modules are connected in series to form an audio generator; and then a comprehensive discriminator is constructed using CNN; Step S32, input the audio data, audio features, and text information into the audio generator, use the text encoder to convert the input text information into a high-dimensional vector representation, process the input text through Transformer and self-attention mechanism, and extract the semantic information and contextual relationship in the text; process the audio features and audio data through the basic generator, map the input high-dimensional feature vector to an audio waveform, and generate an audio signal; use the posterior encoder to obtain the posterior features of the audio signal and estimate the authenticity of the generated audio signal; the residual coupling block uses the residual connection to avoid the gradient vanishing problem, and separates and combines according to the input potential representation; the spectrogram style encoder extracts the spectrogram style information, controls the style and emotional color of the generated audio, and uses CNN to extract deep features so that the model generates diversified audio styles; quantizes the features through the residual vector quantizer, and converts the continuous potential features into discrete representations to reduce the computational complexity; Step S33, build a comprehensive discriminator through a basic discriminator and multiple sub-discriminators of different periods to evaluate the quality of the generated audio; the basic discriminator adopts a deep CNN architecture to process the original input audio signal, is responsible for extracting features from the input signal and making authenticity judgments, and evaluates the input real audio signal to obtain an authenticity score; the input audio is discriminated for multiple periods through a period discriminator, and the audio signal is independently discriminated for different periods to capture features in different frequency ranges; the input audio is transmitted to each period discriminator, processed through a custom convolutional architecture, and the input signal is subjected to a sliding window operation on different time scales to capture subtle changes and patterns, and finally the authenticity score of the corresponding period is returned.

7. The method according to claim 1, characterized in that The voice system in step 4 includes: a host selection module, a segmentation mode module, a speech speed adjustment module, a dubbing content module, a dubbing generation module and a download module; The host selection module is used to select a specific host, and the background will automatically match the reference audio and reference text of the selected host; The segmentation mode module is used to provide multiple segmentation options, including segmentation when four sentences are recognized, segmentation when a period is recognized, and segmentation when the number of words counted reaches 45; The speech rate adjustment module is used to adjust the speed of the synthesized audio, and its speed range is 0.4 to 2 times; The dubbing content module is used to identify the dubbing content, including Chinese characters, punctuation marks, Arabic numerals and time symbols; Generate dubbing module, which is used to divide the dubbing content into multiple short sentences according to the preset segmentation method and speech speed, and then use the text semantic model and speech reasoning model to gradually generate the audio corresponding to each short sentence, and finally integrate the audio corresponding to all short sentences to complete the splicing; The download module is used to download the generated dubbing file to the local computer.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the deep learning-based speech generation method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to perform the deep learning-based speech generation method described in any one of claims 1 to 7.

Images