Voice generation method and device, computer equipment and storage medium
By combining reference speech and speech text representation data for prediction and enhancement processing, the problem that traditional speech synthesis methods cannot generate high-quality speech is solved, and the natural fluency of personalized speech output is achieved.
Patent Information
- Application Number
- CN202411463370.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-02-14
AI Technical Summary
Traditional speech synthesis methods are difficult to generate high-quality speech similar to the reference speech timbre, tone, and emotions, resulting in unnaturally and smoothly synthesized speech.
By receiving the speech generation request sent by the user terminal, the reference speech input speech characterizer is characterized, and the text representation data of the speech text is input to the autoregressive language model for prediction, and then the speech enhancement is performed in the diffusion transformer model, and finally the acoustic speech is performed in the acoustic speech Speech synthesis is performed in the synthesizer.
It realizes the generation of high-quality and personalized voice output, the voice content is consistent with the given text, and the tone, intonation, and emotions are close to the reference voice, improving the natural fluency of voice synthesis.
Smart Images
Figure CN119446115A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech processing technology, and in particular to a speech generation method, apparatus, computer equipment and storage medium. Background Art
[0002] In speech synthesis technology, how to generate a high-quality speech that is similar to the reference speech in timbre, intonation, and emotion, and whose speech content is consistent with the given text, based on the reference speech and speech text provided by the user is a technical problem that needs to be solved urgently.
[0003] Traditional speech synthesis methods usually use a combination of acoustic models and language models to first convert text into phonemes or other speech unit sequences, and then generate speech waveforms through the acoustic model.
[0004] Although this method can synthesize speech that is consistent with the text, it often has large differences from the reference speech in terms of timbre, intonation, emotion, etc., and the synthesized speech is not natural and smooth enough. It can be seen that the traditional speech synthesis method has the problem of being unable to synthesize high-quality speech. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a speech generation method, apparatus, computer device and storage medium to solve the problem that traditional speech synthesis methods cannot synthesize high-quality speech.
[0006] In order to solve the above technical problems, the present application provides a method for generating speech, which adopts the following technical solution:
[0007] Receiving a speech generation request sent by a user terminal, wherein the speech generation request includes a reference speech and a speech text;
[0008] Inputting the reference speech into a speech characterizer to perform speech characterization operation to obtain speech characterization data;
[0009] Inputting the speech text into a text characterizer to perform a text characterization operation to obtain text characterization data;
[0010] Inputting the speech representation data and the text representation data into an autoregressive language model for prediction to obtain speech tag data;
[0011] Inputting the speech tag data into a diffusion transformer model to perform speech enhancement operation to obtain enhanced speech data;
[0012] The enhanced speech data is input into an acoustic speech synthesizer for speech synthesis operation to obtain target synthesized speech.
[0013] Furthermore, after the step of receiving the voice generation request sent by the user terminal, the following steps are also included:
[0014] The reference speech is subjected to denoising processing according to a speech signal analysis algorithm to obtain a high-quality reference speech.
[0015] Furthermore, the step of inputting the reference speech into a speech characterizer to perform speech characterization operation to obtain speech characterization data specifically includes the following steps:
[0016] Inputting the reference speech into a convolutional neural network for feature extraction to obtain local speech features;
[0017] Inputting the local speech features into a recurrent neural network to perform dependency learning operations to obtain long-range dependency relationships;
[0018] The long-range dependency is input into a fully connected layer for transformation and mapping operation to obtain the speech representation data.
[0019] Furthermore, the step of inputting the reference speech into a convolutional neural network for feature extraction to obtain local speech features specifically includes the following steps:
[0020] Acquire the speech segment features obtained by the feature extraction operation;
[0021] Calculating the attention weight of the speech segment feature according to the Bahdanau attention mechanism;
[0022] A weighted operation is performed on the speech segment feature according to the attention weight to obtain the local speech feature.
[0023] Furthermore, the step of inputting the enhanced speech data into an acoustic speech synthesizer for speech synthesis operation to obtain a target synthesized speech specifically includes the following steps:
[0024] Acquire the speech synthesis operation to obtain preliminary synthesized speech;
[0025] The preliminary synthesized speech is subjected to denoising and equalization processing to obtain the target synthesized speech.
[0026] Furthermore, after the step of inputting the enhanced speech data into an acoustic speech synthesizer to perform a speech synthesis operation to obtain a target synthesized speech, the method further includes the following steps:
[0027] Performing acoustic parameter analysis on the reference speech and the target synthesized speech respectively to obtain reference acoustic features and synthesized acoustic features;
[0028] Performing a feature comparison operation on the reference acoustic feature and the synthetic acoustic feature to obtain a difference degree;
[0029] If the difference is less than or equal to the difference threshold, taking the target synthesized speech as the final synthesized speech;
[0030] If the difference is greater than a preset difference threshold, performing a parameter optimization operation on the acoustic speech synthesizer according to the difference to obtain an optimized acoustic speech synthesizer;
[0031] The enhanced speech data is subjected to a speech synthesis operation according to the optimized acoustic speech synthesizer until the output synthesized speech meets the difference threshold, thereby obtaining a final synthesized speech.
[0032] In order to solve the above technical problems, the embodiment of the present application further provides a speech generation device, which adopts the following technical solution:
[0033] A request acquisition module, configured to receive a speech generation request sent by a user terminal, wherein the speech generation request includes a reference speech and a speech text;
[0034] A speech characterization module, used for inputting the reference speech into a speech characterizer to perform speech characterization operation and obtain speech characterization data;
[0035] A text representation module, used for inputting the speech text into a text representation device to perform text representation operation and obtain text representation data;
[0036] A prediction module, used for inputting the speech representation data and the text representation data into an autoregressive language model for prediction operation to obtain speech tag data;
[0037] A speech enhancement module, used for inputting the speech mark data into a diffusion transformer model to perform speech enhancement operation to obtain enhanced speech data;
[0038] The speech synthesis module is used to input the enhanced speech data into an acoustic speech synthesizer to perform speech synthesis operations to obtain target synthesized speech.
[0039] Furthermore, the device also includes:
[0040] The denoising module is used to perform denoising processing on the reference speech according to a speech signal analysis algorithm to obtain high-quality reference speech.
[0041] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0042] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech generation method as described above when executing the computer-readable instructions.
[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0044] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the speech generation method described above are implemented.
[0045] The present application provides a speech generation method, comprising: receiving a speech generation request sent by a user terminal, wherein the speech generation request includes a reference speech and a speech text; inputting the reference speech into a speech characterizer to perform a speech characterization operation to obtain speech characterization data; inputting the speech text into a text characterizer to perform a text characterization operation to obtain text characterization data; inputting the speech characterization data and the text characterization data into an autoregressive language model to perform a prediction operation to obtain speech tag data; inputting the speech tag data into a diffusion transformer model to perform a speech enhancement operation to obtain enhanced speech data; inputting the enhanced speech data into an acoustic speech synthesizer to perform a speech synthesis operation to obtain a target synthesized speech. Compared with the prior art, the present application can generate high-quality, personalized speech output. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0047] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0048] Figure 2 is a flow chart of the implementation of the speech generation method provided in the embodiment of the present application;
[0049] Figure 3 is a structural schematic diagram of a speech generating device provided in an embodiment of the present application;
[0050] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0054] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0055] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0056] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.
[0057] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .
[0058] It should be noted that the speech generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the speech generation device is generally arranged in the server / terminal device.
[0059] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0060] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech generation method according to the present application. The speech generation method comprises: step S201, step S202, step S203, step S204, step S205 and step S206.
[0061] In step S201, a speech generation request sent by a user terminal is received, wherein the speech generation request includes a reference speech and a speech text.
[0062] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.
[0063] In the embodiment of the present application, the user sends a speech generation request through their terminal (such as a smartphone, computer, etc.), which contains two key pieces of information: a reference speech and a speech text. The reference speech refers to the sound or tone that the user wants to imitate, and the speech text is the text content that the user wants to be converted into speech.
[0064] In step S202, the reference speech is input into a speech characterizer to perform a speech characterization operation to obtain speech characterization data.
[0065] In the embodiment of the present application, the speech characterizer is mainly used to identify and analyze different units (such as phonemes, syllables or words) in speech. The result of the speech characterization operation is a set of speech characterization data, which reflects the characteristics of the reference speech, such as pitch, speech rate, intonation, etc.
[0066] In step S203, the speech text is input into a text tokenizer for text tokenization to obtain text tokenization data.
[0067] In the embodiment of the present application, the text characterizer is mainly used to convert the text into a machine-readable format, namely, text characterization data, which contains the semantic information of each word, phrase or sentence in the text, as well as the structural relationship between them.
[0068] In step S204, the speech representation data and the text representation data are input into an autoregressive language model for prediction operation to obtain speech tag data.
[0069] In the embodiment of the present application, the present application combines the speech representation data and the text representation data and inputs them into an autoregressive language model, wherein the autoregressive model is mainly used to predict how to convert the text representation into speech tag data that conforms to the reference speech features. These speech tag data include specific phoneme and syllable tags, and the temporal relationship between them.
[0070] In step S205, the speech tag data is input into the diffusion transformer model for speech enhancement operation to obtain enhanced speech data.
[0071] In the embodiment of the present application, the speech tag data is then input into a diffusion transformer model. The implementation process of the diffusion transformer is to generate high-quality audio by gradually adding noise and learning how to recover the original signal from the noise. In this step, it is used to enhance the sound quality of the speech tag data to make it closer to the characteristics of the reference speech, thereby obtaining enhanced speech data.
[0072] In step S206, the enhanced speech data is input into an acoustic speech synthesizer for speech synthesis operation to obtain a target synthesized speech.
[0073] In the embodiment of the present application, the enhanced speech data is input into an acoustic speech synthesizer. The acoustic speech synthesizer uses the enhanced speech data as input to generate a final target synthesized speech. This target synthesized speech not only contains the content of the speech text, but also imitates the characteristics of the reference speech as much as possible.
[0074] In an embodiment of the present application, a speech generation method is provided, including: receiving a speech generation request sent by a user terminal, wherein the speech generation request includes a reference speech and a speech text; inputting the reference speech into a speech characterizer to perform a speech characterization operation to obtain speech characterization data; inputting the speech text into a text characterizer to perform a text characterization operation to obtain text characterization data; inputting the speech characterization data and the text characterization data into an autoregressive language model to perform a prediction operation to obtain speech tag data; inputting the speech tag data into a diffusion transformer model to perform a speech enhancement operation to obtain enhanced speech data; inputting the enhanced speech data into an acoustic speech synthesizer to perform a speech synthesis operation to obtain a target synthesized speech. Compared with the prior art, the present application can generate high-quality, personalized speech output.
[0075] In some optional implementations of the embodiments of the present application, after the step of receiving the voice generation request sent by the user terminal, the following steps are also included:
[0076] The reference speech is denoised according to the speech signal analysis algorithm to obtain high-quality reference speech.
[0077] In the embodiment of the present application, the speech signal analysis algorithm analyzes and understands the speech content by extracting the features of the speech signal, such as time domain features (variance, average energy, etc.) and frequency domain features (FFT, spectral density, etc.). This feature extraction process provides the necessary technical support for the subsequent speech denoising process.
[0078] In the embodiment of the present application, the denoising process includes noise signal acquisition, preprocessing, feature extraction, noise identification and modeling, and denoising algorithm application, wherein:
[0079] 1) Noise signal acquisition: Collecting the original speech signal containing noise is the first step of denoising;
[0080] 2) Preprocessing: digitize and sample the collected speech signals for subsequent digital signal processing (DSP), convert them to the DSP environment, and prepare for denoising;
[0081] 3) Feature extraction: Use speech signal analysis algorithms to extract the features of speech signals, which will be used for subsequent noise identification and removal;
[0082] 4) Noise identification and modeling: By analyzing the spectral characteristics of the speech signal, the position of the noise component on the spectrum is identified, and the noise is modeled using filters (such as band-stop filters, high-pass filters, etc.) or machine learning models (such as deep neural networks);
[0083] 5) Application of denoising algorithm: Apply frequency domain-based denoising algorithm: By analyzing the spectral characteristics of the speech signal, find the position of the noise component on the spectrum, and then use filters and other methods to eliminate the noise. This method has low computational complexity, but may not be suitable for non-stationary noise and nonlinear noise. Apply time domain-based denoising algorithm: By performing short-time Fourier transform (STFT) on the speech signal, extract the characteristic frequency components of the signal, and then eliminate the noise based on these characteristic frequency components. This method has a good processing effect on non-stationary noise and nonlinear noise, but the computational complexity is high. Apply neural network-based denoising algorithm: By training a large amount of noisy speech data, learn the ability to distinguish useful signals from noise, and then apply this ability to the actual speech denoising task. This method has strong adaptability to complex noise, but requires a lot of training data and computing resources.
[0084] In the embodiment of the present application, the noise component in the original speech signal can be significantly reduced through the above-mentioned denoising process, thereby obtaining a high-quality reference speech.
[0085] In some optional implementations of the embodiments of the present application, the step of inputting the reference speech into the speech characterizer to perform the speech characterization operation to obtain the speech characterization data specifically includes the following steps:
[0086] Input the reference speech into the convolutional neural network for feature extraction to obtain local speech features;
[0087] Input local speech features into a recurrent neural network for dependency learning operations to obtain long-range dependencies;
[0088] The long-range dependency is input into the fully connected layer for transformation and mapping operations to obtain speech representation data.
[0089] In an embodiment of the present application, a convolutional neural network (CNN) is a neural network that is particularly suitable for processing data with a grid topology (such as images and speech signals). In speech processing, CNN can capture local features in speech signals, such as phonemes, syllables, etc., by learning convolution kernels (or filters). Among them, the input layer of CNN usually accepts representations such as spectrograms or mel-spectrograms of speech signals, which can better reflect the spectral characteristics of speech. Through multi-layer convolution and pooling operations, CNN can gradually extract higher-level local features. After processing by CNN, the present application obtains a series of local speech features. These features are usually expressed as feature vectors or feature maps, which capture local structures and patterns in speech signals.
[0090] In the embodiment of the present application, a recurrent neural network (RNN) is a neural network that can process sequence data, which captures the temporal dependency in the sequence by introducing a recurrent connection. In speech processing, RNN can learn the temporal features in speech signals, such as the combination of syllables, changes in intonation, etc.
[0091] In the embodiment of the present application, the local speech features are input into the RNN, and the RNN will capture the temporal dependencies between these features by learning its internal recurrent connections. The hidden layer state of the RNN can be regarded as encoding the current position and previous position of the input sequence, which enables the RNN to process input sequences of arbitrary length and capture the long-range dependencies therein, which reflect the temporal associations and dependencies between different parts of the speech signal.
[0092] In an embodiment of the present application, the fully connected layer is generally used to map the long-range dependencies output by the RNN into a higher-level representation space to form speech representation data.
[0093] In the embodiment of the present application, the long-range dependency is input into the fully connected layer, which transforms and maps these relationships by learning their weights and biases. The purpose of the transformation mapping is to convert the long-range dependency into more expressive and discriminative speech representation data.
[0094] In the embodiment of the present application, after the processing of the fully connected layer, the present application obtains the final speech representation data. These data are abstract representations of the original speech signal, which capture the key information and features in the speech signal. These speech representation data can be used for various speech processing tasks, such as speech recognition, speech synthesis, emotion recognition, etc.
[0095] In an embodiment of the present application, by inputting the reference speech into a convolutional neural network for feature extraction, then inputting these local speech features into a recurrent neural network for dependency learning, and inputting the obtained long-range dependency into a fully connected layer for transformation mapping, the present application can obtain speech representation data with rich information and expressiveness.
[0096] In some optional implementations of the embodiments of the present application, the step of inputting the reference speech into a convolutional neural network for feature extraction to obtain local speech features specifically includes the following steps:
[0097] Acquire the speech segment features obtained by the feature extraction operation;
[0098] Calculating the attention weight of the speech segment feature according to the Bahdanau attention mechanism;
[0099] A weighted operation is performed on the speech segment feature according to the attention weight to obtain the local speech feature.
[0100] In the embodiment of the present application, the Bahdanau attention mechanism is a technique commonly used in sequence-to-sequence (Seq2Seq) models, which allows the model to dynamically focus on different parts of the input sequence at each step of generating the output sequence. The mechanism determines which elements are most important for generating the current output by calculating the correlation score between each element in the input sequence and the current output state.
[0101] In an embodiment of the present application, the Bahdanau attention mechanism is applied to speech segment features. For each speech segment feature, the present application calculates the correlation score between it and the current processing state (which may be the hidden state of the recurrent neural network RNN or the long short-term memory network LSTM). These correlation scores are normalized by the softmax function to obtain the attention weight of each speech segment feature.
[0102] In the embodiment of the present application, the weighting operation can be performed by performing a weighting operation on the speech segment features according to the calculated attention weights. Specifically, each speech segment feature is multiplied by its corresponding attention weight to obtain a weighted feature vector.
[0103] In an embodiment of the present application, by acquiring speech segment features obtained by feature extraction operations, calculating the attention weights of these features according to the Bahdanau attention mechanism, and performing weighted operations on the speech segment features according to these weights, we can obtain local speech features that are more focused on useful information.
[0104] In some optional implementations of the embodiments of the present application, the step of inputting the enhanced speech data into an acoustic speech synthesizer for speech synthesis to obtain a target synthesized speech specifically includes the following steps:
[0105] Acquire the speech synthesis operation to obtain preliminary synthesized speech;
[0106] The preliminary synthesized speech is subjected to denoising and equalization processing to obtain the target synthesized speech.
[0107] In the embodiment of the present application, the denoising process can use a variety of methods, such as spectrum subtraction, Wiener filtering, noise estimation and suppression, etc. Through the denoising process, the noise interference in the preliminary synthesized speech can be reduced and the clarity of the speech can be improved.
[0108] In an embodiment of the present application, equalization processing makes the speech sound more natural and comfortable by adjusting the spectrum envelope of the speech signal. The equalization processing method may be a spectrum equalizer, dynamic range compression, etc.
[0109] In the embodiment of the present application, by obtaining the preliminary synthesized speech obtained by the speech synthesis operation and performing denoising and equalization processing on it, we can obtain a clearer, more natural and high-quality target synthesized speech.
[0110] In some optional implementations of the embodiments of the present application, after the step of inputting the enhanced speech data into the acoustic speech synthesizer for speech synthesis operation to obtain the target synthesized speech, the following steps are also included:
[0111] Perform acoustic parameter analysis on the reference speech and the target synthesized speech respectively to obtain reference acoustic features and synthesized acoustic features;
[0112] Perform feature comparison operation on the reference acoustic feature and the synthetic acoustic feature to obtain the difference degree;
[0113] If the difference is less than or equal to the difference threshold, the target synthesized speech is used as the final synthesized speech;
[0114] If the difference is greater than a preset difference threshold, performing parameter optimization operation on the acoustic speech synthesizer according to the difference to obtain an optimized acoustic speech synthesizer;
[0115] The enhanced speech data is subjected to speech synthesis operation according to the optimized acoustic speech synthesizer until the output synthesized speech meets the difference threshold, thereby obtaining the final synthesized speech.
[0116] In the embodiment of the present application, acoustic parameter analysis is a process of quantitatively describing speech signals, which extracts key acoustic features in speech signals, such as fundamental frequency (F0), formant, energy, speech rate, intonation, etc. These acoustic features can reflect the characteristics of speech such as sound quality, timbre, emotion, etc.
[0117] In the embodiment of the present application, feature comparison is a process of comparing the reference acoustic features with the synthesized acoustic features one by one, and aims to find the difference between the two sets of features to evaluate the similarity between the target synthesized speech and the reference speech in terms of acoustic characteristics.
[0118] In the embodiment of the present application, the difference is a quantitative index to measure the difference between the reference acoustic features and the synthesized acoustic features. It can be obtained by calculating the Euclidean distance, Manhattan distance, cosine similarity and other measurement methods between two sets of feature vectors. The smaller the difference, the more similar the target synthesized speech is to the reference speech in terms of acoustic characteristics.
[0119] In the embodiment of the present application, by performing acoustic parameter analysis, feature comparison, difference calculation, and parameter optimization on the reference speech and the target synthesized speech, we can obtain a high-quality final synthesized speech. This process not only improves the quality of speech synthesis, but also enhances the generalization ability and performance of the synthesizer.
[0120] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0121] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0123] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0124] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech generating device, and the embodiment of the device is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0125] like Figure 3 As shown, the speech generating device 200 of the embodiment of the present application includes:
[0126] The request acquisition module 210 is used to receive a speech generation request sent by a user terminal, wherein the speech generation request includes a reference speech and a speech text;
[0127] The speech characterization module 220 is used to input the reference speech into the speech characterizer to perform speech characterization operation and obtain speech characterization data;
[0128] The text representation module 230 is used to input the speech text into the text representation device to perform text representation operation and obtain text representation data;
[0129] A prediction module 240 is used to input the speech representation data and the text representation data into the autoregressive language model for prediction operation to obtain speech tag data;
[0130] The speech enhancement module 250 is used to input the speech tag data into the diffusion transformer model to perform speech enhancement operation to obtain enhanced speech data;
[0131] The speech synthesis module 260 is used to input the enhanced speech data into the acoustic speech synthesizer for speech synthesis operation to obtain the target synthesized speech.
[0132] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.
[0133] In the embodiment of the present application, the user sends a speech generation request through their terminal (such as a smartphone, computer, etc.), which contains two key pieces of information: a reference speech and a speech text. The reference speech refers to the sound or tone that the user wants to imitate, and the speech text is the text content that the user wants to be converted into speech.
[0134] In the embodiment of the present application, the speech characterizer is mainly used to identify and analyze different units (such as phonemes, syllables or words) in speech. The result of the speech characterization operation is a set of speech characterization data, which reflects the characteristics of the reference speech, such as pitch, speech rate, intonation, etc.
[0135] In the embodiment of the present application, the text characterizer is mainly used to convert the text into a machine-readable format, namely, text characterization data, which contains the semantic information of each word, phrase or sentence in the text, as well as the structural relationship between them.
[0136] In the embodiment of the present application, the present application combines the speech representation data and the text representation data and inputs them into an autoregressive language model, wherein the autoregressive model is mainly used to predict how to convert the text representation into speech tag data that conforms to the reference speech features. These speech tag data include specific phoneme and syllable tags, and the temporal relationship between them.
[0137] In the embodiment of the present application, the speech tag data is then input into a diffusion transformer model. The implementation process of the diffusion transformer is to generate high-quality audio by gradually adding noise and learning how to recover the original signal from the noise. In this step, it is used to enhance the sound quality of the speech tag data to make it closer to the characteristics of the reference speech, thereby obtaining enhanced speech data.
[0138] In the embodiment of the present application, the enhanced speech data is input into an acoustic speech synthesizer. The acoustic speech synthesizer uses the enhanced speech data as input to generate a final target synthesized speech. This target synthesized speech not only contains the content of the speech text, but also imitates the characteristics of the reference speech as much as possible.
[0139] In an embodiment of the present application, a speech generation device 200 is provided, including: a request acquisition module 210, for receiving a speech generation request sent by a user terminal, wherein the speech generation request includes a reference speech and a speech text; a speech characterization module 220, for inputting the reference speech into a speech characterizer for speech characterization operation to obtain speech characterization data; a text characterization module 230, for inputting the speech text into a text characterizer for text characterization operation to obtain text characterization data; a prediction module 240, for inputting the speech characterization data and the text characterization data into an autoregressive language model for prediction operation to obtain speech tag data; a speech enhancement module 250, for inputting the speech tag data into a diffusion transformer model for speech enhancement operation to obtain enhanced speech data; a speech synthesis module 260, for inputting the enhanced speech data into an acoustic speech synthesizer for speech synthesis operation to obtain a target synthesized speech. Compared with the prior art, the present application can generate high-quality, personalized speech output.
[0140] In some optional implementations of the embodiments of the present application, the speech generating device 200 further includes:
[0141] The denoising module is used to perform denoising on the reference speech according to the speech signal analysis algorithm to obtain high-quality reference speech.
[0142] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.
[0143] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0144] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0145] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the speech generation method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.
[0146] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the speech generation method.
[0147] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.
[0148] The computer device provided by this application can generate high-quality, personalized voice output.
[0149] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech generation method as described above.
[0150] The computer-readable storage medium provided in this application can generate high-quality, personalized voice output.
[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0152] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A speech generation method, characterized in that: The steps include: Receiving a speech generation request sent by a user terminal, wherein the speech generation request includes a reference speech and a speech text; Inputting the reference speech into a speech characterizer to perform speech characterization operation to obtain speech characterization data; Inputting the speech text into a text characterizer to perform a text characterization operation to obtain text characterization data; Inputting the speech representation data and the text representation data into an autoregressive language model for prediction to obtain speech tag data; Inputting the speech tag data into a diffusion transformer model to perform speech enhancement operation to obtain enhanced speech data; The enhanced speech data is input into an acoustic speech synthesizer for speech synthesis operation to obtain target synthesized speech.
2. The speech generation method according to claim 1, characterized in that: After the step of receiving the voice generation request sent by the user terminal, the method further includes the following steps: The reference speech is subjected to denoising processing according to a speech signal analysis algorithm to obtain a high-quality reference speech.
3. The speech generation method according to claim 1, characterized in that: The step of inputting the reference speech into a speech characterizer to perform a speech characterization operation to obtain speech characterization data specifically comprises the following steps: Inputting the reference speech into a convolutional neural network for feature extraction to obtain local speech features; Inputting the local speech features into a recurrent neural network to perform dependency learning operations to obtain long-range dependency relationships; The long-range dependency is input into a fully connected layer for transformation and mapping operation to obtain the speech representation data.
4. The speech generation method according to claim 3, characterized in that: The step of inputting the reference speech into a convolutional neural network for feature extraction to obtain local speech features specifically includes the following steps: Acquire the speech segment features obtained by the feature extraction operation; Calculating the attention weight of the speech segment feature according to the Bahdanau attention mechanism; A weighted operation is performed on the speech segment feature according to the attention weight to obtain the local speech feature.
5. The speech generation method according to claim 1, characterized in that: The step of inputting the enhanced speech data into an acoustic speech synthesizer for speech synthesis operation to obtain a target synthesized speech specifically comprises the following steps: Acquire the speech synthesis operation to obtain preliminary synthesized speech; The preliminary synthesized speech is subjected to denoising and equalization processing to obtain the target synthesized speech.
6. The speech generation method according to claim 1, characterized in that: After the step of inputting the enhanced speech data into an acoustic speech synthesizer to perform speech synthesis operation to obtain a target synthesized speech, the method further includes the following steps: Performing acoustic parameter analysis on the reference speech and the target synthesized speech respectively to obtain reference acoustic features and synthesized acoustic features; Performing a feature comparison operation on the reference acoustic feature and the synthetic acoustic feature to obtain a difference degree; If the difference is less than or equal to the difference threshold, taking the target synthesized speech as the final synthesized speech; If the difference is greater than a preset difference threshold, performing a parameter optimization operation on the acoustic speech synthesizer according to the difference to obtain an optimized acoustic speech synthesizer; The enhanced speech data is subjected to a speech synthesis operation according to the optimized acoustic speech synthesizer until the output synthesized speech meets the difference threshold, thereby obtaining a final synthesized speech.
7. A speech generating device, characterized in that: include: A request acquisition module, configured to receive a speech generation request sent by a user terminal, wherein the speech generation request includes a reference speech and a speech text; A speech characterization module, used for inputting the reference speech into a speech characterizer to perform speech characterization operation and obtain speech characterization data; A text representation module, used for inputting the speech text into a text representation device to perform text representation operation and obtain text representation data; A prediction module, used for inputting the speech representation data and the text representation data into an autoregressive language model for prediction operation to obtain speech tag data; A speech enhancement module, used for inputting the speech mark data into a diffusion transformer model to perform speech enhancement operation to obtain enhanced speech data; The speech synthesis module is used to input the enhanced speech data into an acoustic speech synthesizer to perform speech synthesis operations to obtain target synthesized speech.
8. The speech generating device according to claim 7, characterized in that: The device also includes: The denoising module is used to perform denoising processing on the reference speech according to a speech signal analysis algorithm to obtain high-quality reference speech.
9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the speech generation method according to any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech generation method according to any one of claims 1 to 6 are implemented.