Multi-module collaborative voice generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a multi-module collaborative voice generation method, which comprises the following steps: acquiring a reference text and reference voice, converting the reference text into a phoneme sequence and generating potential features, converting the reference voice into a Mel spectrogram and extracting style features, and extracting fundamental frequency information. A phoneme sequence and a Mel spectrogram are input into a text alignment module to generate alignment information, voice duration is predicted based on the alignment information and potential features, fundamental frequency of voice is predicted through fundamental frequency information, the predicted duration, fundamental frequency and style features are input into a decoder to generate a reconstructed Mel spectrogram, and finally the reconstructed Mel spectrogram is converted into voice audio data through a vocoder. According to the method, the generated voice can better reflect the natural rhythm and tone change of the text content by combining accurate control on the voice duration, the fundamental frequency and the style characteristics, and the rhythm and expressive force of the voice are enhanced, so that the naturalness and the fidelity of voice synthesis are improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a multi-module collaborative speech generation method, device, equipment and storage medium. Background Art

[0002] With the rapid development of deep learning in recent years, speech synthesis technology has made significant progress. Through advanced neural network models, the naturalness and sound quality of synthesized speech have been greatly improved, and the generated speech is getting closer and closer to real human speech. However, existing technologies still have significant deficiencies in the generation of expressive speech, especially in capturing paralinguistic information such as rhythm and spectral features. The expressiveness of synthesized speech is still far behind that of natural human speech.

[0003] Text-to-speech (TTS) is essentially a one-to-many mapping problem. The same text can generate different speech styles based on the context, emotional expression, and the speaker's personal habits. Existing speech synthesis systems usually show certain limitations in processing the expression of different emotions or tones, and cannot accurately capture the unique language habits and style of each speaker. As a result, the generated speech is clear and natural, but still has obvious deficiencies in emotional expression and personalized performance.

[0004] In the financial field, speech synthesis technology has a wide range of application scenarios, such as financial customer service systems, intelligent voice assistants, automatic voice notification systems, etc. Financial institutions usually need to respond to diverse customer needs and provide personalized voice services, so the emotional expression and stylization of synthesized speech are crucial.

[0005] However, existing technologies are still not ideal in accurately modeling speech emotions, and it is difficult to generate emotionally rich and delicate speech. When dealing with complex contexts or emotional changes, the emotional connection between speech and text content is not close enough, and the generated speech may deviate from the actual emotion or intention expressed. Summary of the invention

[0006] The main purpose of the present invention is to provide a multi-module collaborative speech generation method, device, equipment and storage medium, aiming to solve the technical problem that it is difficult to accurately control the rhythm, fundamental frequency and duration of the generated speech in speech synthesis in the prior art, resulting in the generated speech being obviously insufficient in naturalness and expressiveness.

[0007] To achieve the above object, the present invention provides a multi-module collaborative speech generation method, comprising:

[0008] Obtain reference text and reference speech;

[0009] Converting the reference text into a phoneme sequence, and inputting the phoneme sequence into a text encoding module to generate latent features;

[0010] Convert the reference speech into a Mel spectrogram, input the Mel spectrogram into a style encoding module, and extract style features by the style encoding module;

[0011] Extracting fundamental frequency information from the Mel spectrogram by a fundamental frequency extraction module;

[0012] Input the phoneme sequence and the Mel spectrogram into a text alignment module, and the text alignment module generates alignment information;

[0013] Based on the alignment information and the latent features, predicting the duration of the generated speech by a duration prediction module;

[0014] Predicting the fundamental frequency of the generated speech based on the fundamental frequency information by a fundamental frequency prediction module;

[0015] Inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to generate a reconstructed Mel spectrogram;

[0016] The reconstructed Mel spectrogram is converted into speech audio data using a vocoder.

[0017] Furthermore, to achieve the above-mentioned object, the present invention provides a multi-module collaborative speech generation device, comprising:

[0018] A data acquisition module, used to acquire reference text and reference speech;

[0019] A text encoding module, used for converting the reference text into a phoneme sequence, and inputting the phoneme sequence into the text encoding module to generate latent features;

[0020] A style encoding module, used for converting the reference speech into a Mel spectrogram, and inputting the Mel spectrogram into the style encoding module, and the style encoding module extracts style features;

[0021] A fundamental frequency extraction module, used for extracting fundamental frequency information from the Mel spectrogram through the fundamental frequency extraction module;

[0022] A text alignment module, used for inputting the phoneme sequence and the Mel spectrogram into the text alignment module, and the text alignment module generates alignment information;

[0023] A duration prediction module, configured to predict the duration of the generated speech through the duration prediction module based on the alignment information and the latent features;

[0024] A fundamental frequency prediction module, configured to predict the fundamental frequency of the generated speech based on the fundamental frequency information through the fundamental frequency prediction module;

[0025] A decoder module, used for inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to generate a reconstructed Mel spectrogram;

[0026] The vocoder module is used to convert the reconstructed Mel spectrogram into speech audio data using a vocoder.

[0027] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a multi-module collaborative speech generation program stored in the memory and executable on the processor, and the multi-module collaborative speech generation program, when executed by the processor, implements the steps of the multi-module collaborative speech generation method as described above.

[0028] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a multi-module collaborative speech generation program is stored, and when the multi-module collaborative speech generation program is executed by a processor, the steps of the multi-module collaborative speech generation method as described above are implemented.

[0029] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a multi-module collaborative speech generation method, which obtains reference text and reference speech, converts the reference text into a phoneme sequence and generates potential features, converts the reference speech into a Mel spectrogram and extracts style features, extracts fundamental frequency information, inputs the phoneme sequence and the Mel spectrogram into a text alignment module to generate alignment information, predicts speech duration based on alignment information and potential features, predicts the fundamental frequency of speech through fundamental frequency information, inputs the predicted duration, fundamental frequency and style features into a decoder to generate a reconstructed Mel spectrogram, and finally converts the reconstructed Mel spectrogram into speech audio data through a vocoder. The present invention combines precise control of speech duration, fundamental frequency and style features, so that the generated speech can better reflect the natural rhythm and pitch changes of the text content, and enhances the rhythm and expressiveness of the speech, thereby improving the naturalness and fidelity of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0031] Figure 1 A schematic diagram of an application environment of a multi-module collaborative speech generation method according to an embodiment of the present invention;

[0032] Figure 2 A flowchart of an embodiment of a multi-module collaborative speech generation method of the present invention;

[0033] Figure 3A schematic diagram of functional modules of a preferred embodiment of a multi-module collaborative speech generation device of the present invention;

[0034] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0035] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0036] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0037] The multi-module collaborative speech generation method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can obtain reference text and reference speech through the user end, convert the reference text into a phoneme sequence and generate potential features, convert the reference speech into a Mel spectrogram and extract style features, extract fundamental frequency information, input the phoneme sequence and the Mel spectrogram into a text alignment module to generate alignment information, predict the speech duration based on the alignment information and potential features, predict the fundamental frequency of the speech through the fundamental frequency information, input the predicted duration, fundamental frequency and style features into a decoder to generate a reconstructed Mel spectrogram, and finally convert the reconstructed Mel spectrogram into speech audio data through a vocoder. The present invention combines the precise control of speech duration, fundamental frequency and style features to enable the generated speech to better reflect the natural rhythm and pitch changes of the text content, and enhance the rhythm and expressiveness of the speech, thereby improving the naturalness and fidelity of speech synthesis. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0038] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a multi-module collaborative speech generation method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0039] like Figure 2 As shown, the multi-module collaborative speech generation method proposed in the present invention includes the following steps:

[0040] S10, obtaining reference text and reference speech;

[0041] In this embodiment, the reference text refers to natural language text, which usually contains letters, numbers, punctuation marks and possible non-language symbols. This text will be used for subsequent speech synthesis. The reference text can come from user input, predefined files, database query results, network crawling or API calls, etc. The acquisition of text is the first step in subsequent speech generation to ensure that the system can generate corresponding speech based on the text content. The reference text can be obtained in the following ways:

[0042] Database query: Extract specific text data from databases in financial systems, such as customer problem records, contract texts, or financial reports.

[0043] User input: Customers enter queries or questions through the interactive interface, and the system saves them in text form.

[0044] API call: Get text data from external systems through API, such as market analysis or stock price descriptions from third-party information service providers.

[0045] File reading: Extracts stored documents or reports from the file system on the server, such as PDF files, and converts the content into text format.

[0046] Web crawling: Use web crawlers to capture relevant financial news or announcement text content from public websites for subsequent analysis and speech generation.

[0047] Data can be read through programming languages ​​such as Python and Java combined with corresponding libraries, such as Python's pandas to read Excel, SQLAlchemy for database interaction, and requests for API requests.

[0048] Reference speech refers to a segment of audio signal, which is usually used for subsequent style feature extraction and fundamental frequency information analysis. Reference speech can be a pre-recorded audio file (such as WAV, MP3 format) or a real-time recorded user voice. The audio usually contains the target pitch, style features and other speaker-related information, which is used as a reference for speech synthesis. Reference speech can be obtained in the following ways:

[0049] Audio file reading: Read existing audio files from the file system, common formats such as WAV, MP3, etc.

[0050] Real-time recording: The user's real-time voice input is obtained through a microphone and other devices. Real-time recording requires the support of audio acquisition hardware and its driver, combined with the audio processing interface at the software level to capture sound.

[0051] API call: voice samples are obtained through cloud-based voice service APIs (such as Google Cloud SpeechAPI, AWS Polly). The system can call the voice files provided by these services as reference audio.

[0052] Audio data can be processed using Python libraries such as Librosa and Pydub to read and analyze audio files. For real-time recording, libraries such as PyAudio are usually used to obtain audio streams from microphones.

[0053] For example, when a customer inquires about product information through online banking services, the system can obtain text information related to the product (such as interest rates, loan conditions, etc.) from the database, and extract reference voices from historical customer service recordings to guide the generation of personalized voice answers. By obtaining text content and reference voices, the system can automatically generate a voice synthesis response that matches the customer's voice style and has accurate content. This not only improves the customer experience, but also speeds up the response efficiency of intelligent customer service.

[0054] By obtaining reference text and reference speech, the necessary input basis can be provided for subsequent speech generation. The reference text provides a textual basis for the generation of speech content, while the reference speech provides a basis for the style and fundamental frequency of the generated speech, making the generated speech more natural and personalized, and improving the accuracy and expressiveness of speech synthesis.

[0055] S20, converting the reference text into a phoneme sequence, and inputting the phoneme sequence into a text encoding module to generate latent features;

[0056] In this embodiment, phonemes are the basic pronunciation units of speech, and the reference text needs to be decomposed into phoneme sequences first. Converting natural language text (including letters, words, and sentences) into phoneme sequences is a key step in speech synthesis. This process usually includes the following sub-steps:

[0057] Text preprocessing: remove punctuation, spaces, or other non-language symbols from the text.

[0058] Word segmentation: Breaking down text into lexical units facilitates subsequent phoneme conversion.

[0059] Phoneme mapping: Use a specific phoneme dictionary or language model (such as CMU Pronouncing Dictionary) to map each word to its corresponding phoneme sequence. For example, the word "hello" will be mapped to the phonemes "HHAH0 L OW1".

[0060] You can use a natural language processing (NLP) toolkit, such as Phonemizer or CMU Sphinx, to convert text into a phoneme sequence. In the financial field, the text may come from financial reports, policy statements, or product introductions. The system needs to convert the words in these texts into phoneme sequences so that the relevant content can be accurately expressed when synthesizing speech. This can be done by combining Python with an NLP toolkit.

[0061] After being generated, the phoneme sequence is passed as input to the text encoding module. The task of the text encoding module is to convert the phoneme sequence into latent features, which are semantic vector representations of the generated speech content. Subsequent steps rely on the latent features to control the content and pronunciation of the speech.

[0062] The text encoding module usually uses a deep learning model, such as RNN (recurrent neural network), Transformer or other language models. The phoneme sequence is mapped to a high-dimensional vector space through an embedding layer to form latent features that are used to capture the semantic information of the text. The generated latent features are compact expressions of the text, and the subsequent decoder will generate speech that conforms to the phoneme content based on these features.

[0063] The text encoding module can be trained using deep learning frameworks (such as TensorFlow, PyTorch). During training, the model learns based on a large number of annotated phoneme sequences and voice pairs, and finally converts the input phoneme sequence into latent features in the inference phase. The latent features can retain the semantic and phonetic information of the text and are used to generate natural speech output.

[0064] By converting the reference text into a phoneme sequence and generating latent features, the system can extract clear speech generation instructions from the input text. The generated latent features provide semantic information and phoneme control for subsequent speech synthesis, ensuring that the speech output can accurately correspond to the input text and guaranteeing natural conversion from text to speech.

[0065] S30, converting the reference speech into a Mel spectrogram, and inputting the Mel spectrogram into a style encoding module, and extracting style features by the style encoding module;

[0066] In this embodiment, the Mel spectrogram is a time-frequency diagram obtained after the audio signal is processed by short-time Fourier transform (STFT) and Mel filter. It is a low-dimensional representation of the speech signal, which can retain the key features of the speech signal (such as pitch, tone and frequency information), and facilitate subsequent feature extraction and speech generation. The conversion process includes the following sub-steps:

[0067] Preprocessing: The reference speech signal is preprocessed, including denoising, normalization, and other general audio processing steps to improve signal quality.

[0068] Short-time Fourier transform (STFT): Perform a short-time Fourier transform on the preprocessed speech signal to convert the time domain signal into a frequency domain signal. STFT divides the audio into several time periods, performs a Fourier transform on the signal in each time period, and generates a spectrum.

[0069] Apply Mel filter: The spectrum result of STFT is processed through Mel filter and converted into spectrum representation on Mel scale. This process simulates the human ear's perception of frequency into the filter and generates Mel spectrogram.

[0070] An audio processing library such as Librosa or Pydub can be used to perform STFT calculations and generate Mel spectrograms. The application of Mel filters can be handled directly by library functions. This step requires efficient audio processing capabilities to generate high-quality Mel spectrograms to ensure the accuracy of subsequent feature extraction. In the financial field, the reference speech may be a customer conversation record or a telephone banking recording, and the generated Mel spectrogram will be used to capture the characteristics of these speeches.

[0071] The style encoding module is used to extract the speaker's style features from the generated Mel spectrogram. These features can include intonation, rhythm, speaking speed, pitch, etc. Style features are important factors affecting the expressiveness of speech in the speech synthesis process and are used to control the emotion and expression of the synthesized speech.

[0072] The style encoding module usually extracts these features through a deep learning model (such as a convolutional neural network (CNN) or a long short-term memory (LSTM) network). The model captures the style elements of speech from the time series of the Mel spectrogram. The extracted style features are usually represented as high-dimensional vectors, which are used to adjust the generated speech style during subsequent speech synthesis. Features may include different dimensions such as rhythm, speaking speed, pitch, emotion, etc.

[0073] The style encoding module can be implemented by training a neural network model. Deep learning frameworks (such as TensorFlow and PyTorch) are often used to train these models, inputting Mel spectrograms and outputting style feature vectors. In financial applications, this can help generate synthesized speech that is consistent with the customer's voice style, ensuring personalized speech output, which is suitable for scenarios such as customer service and financial report generation.

[0074] Example description: In the financial field, banks can extract voice style features from customer voice recordings. The system converts the recordings into Mel spectrograms and captures style information such as intonation, speech speed, and emotion through the style encoding module. This information can be applied to the intelligent voice customer service system. After extracting the customer's voice style, the system synthesizes voice answers that are consistent with their style. For example, when a customer consults about a loan product, the system can generate a synthesized voice that matches the customer's voice rhythm and tone, providing the customer with a more natural service experience.

[0075] By converting the reference speech into a Mel spectrogram and extracting style features, the system can effectively capture the rhythm, pitch, speaking speed and other style characteristics of the speech from the input speech. These style features provide an important basis for subsequent speech synthesis, so that the generated speech can be consistent with the reference speech in timbre and rhythm, improving the naturalness and personalized performance of speech synthesis, which is particularly suitable for customer service scenarios and voice broadcasting tasks in the financial field.

[0076] S40, extracting fundamental frequency information from the Mel spectrogram by a fundamental frequency extraction module;

[0077] In this embodiment, fundamental frequency (F0) is the most basic information in the speech signal that reflects the frequency of vocal cord vibration, and is generally considered to be the "pitch" of speech. Fundamental frequency information directly affects the intonation and rhythm of speech, and is an important parameter for controlling the naturalness and expressiveness of speech. Extracting fundamental frequency information is a key step in speech synthesis, and can effectively reflect the pitch changes of the speaker. The extraction of fundamental frequency is usually achieved by analyzing the spectral envelope of speech from the Mel spectrogram. Common fundamental frequency extraction methods include autocorrelation method, harmonic analysis, dynamic programming fundamental frequency extraction (Dynamic Programming F0 estimation, DIO), etc. For example:

[0078] Autocorrelation method: By analyzing the time domain or frequency domain signal, the peak value in the periodic signal is detected to determine the fundamental frequency.

[0079] Harmonic analysis: Extract the fundamental frequency by analyzing the harmonic structure in the frequency components.

[0080] DIO algorithm: It is an advanced fundamental frequency estimation method, widely used in high-precision speech synthesis systems.

[0081] The Mel spectrogram generated by STFT contains the time-frequency characteristics of speech. The system analyzes the Mel spectrogram through the fundamental frequency extraction module to determine the pitch changes within the frequency range. The specific fundamental frequency extraction process can be implemented through an audio processing library (such as Librosa, Praat or OpenSMILE). First, the Mel spectrogram is processed to detect the frequency peaks of vocal cord vibration and extract these peaks as fundamental frequency data.

[0082] The fundamental frequency extraction module extracts the pitch information related to the speaker from the Mel spectrogram and generates a fundamental frequency (F0) sequence. This fundamental frequency information will be used to control the pitch and rhythm of the generated speech to ensure the naturalness of the synthesized speech. The fundamental frequency information can also be used for subsequent style control and duration prediction.

[0083] The fundamental frequency extraction module can be implemented through algorithms or deep learning models. By analyzing the frequency domain of the input Mel spectrogram, the fundamental frequency information at each moment is extracted to form a fundamental frequency time series. This module can be used in conjunction with other modules in the speech synthesis system (such as the duration prediction module and decoder) to generate accurate speech signals.

[0084] Example description: In intelligent voice applications in the financial field, the fundamental frequency extraction module can help generate voice with natural pitch changes. For example, the system can generate voice answers based on questions raised by customers (such as "What is the current loan interest rate?"). The fundamental frequency extraction module extracts pitch information from the reference voice to ensure that the generated voice matches the style of the customer's conversation in intonation. This technology is particularly important in telephone banking and customer service systems, helping the generated voice to be more natural and friendly, thereby improving the customer experience. In addition, in the automatic reading system of financial reports, the fundamental frequency extraction module can ensure that the voice broadcast of the report content conforms to a professional and standardized voice style.

[0085] By extracting fundamental frequency information from the Mel spectrogram through the fundamental frequency extraction module, the system can accurately obtain the pitch and rhythm characteristics of the reference speech. This information is crucial in the speech synthesis process, ensuring that the generated speech is consistent with the reference speech in terms of pitch, rhythm, etc., improving the naturalness and expressiveness of the synthesized speech. In addition, accurate fundamental frequency extraction can further optimize the emotional expression of the speech, making the generated speech more vivid and expressive.

[0086] S50, inputting the phoneme sequence and the Mel spectrogram into a text alignment module, and the text alignment module generates alignment information;

[0087] In this embodiment, the phoneme sequence is a sequence of speech pronunciation units after the reference text is converted by phonemes, and the Mel spectrogram is a spectrum representation of the reference speech. These two inputs are the core components of content and acoustic features in the speech generation process. The purpose of inputting the phoneme sequence and the Mel spectrogram into the text alignment module is to establish a corresponding relationship between the phonemes and the speech signals.

[0088] Phoneme sequences and Mel spectrograms are usually represented in the form of time series. The length of the phoneme sequence may be different from the duration of the speech signal, so it is necessary to align the phoneme sequence and Mel spectrogram at the frame level through an alignment algorithm. This step can be achieved using an alignment algorithm based on dynamic time warping (DTW) or a neural network (such as an alignment model based on an attention mechanism). For example:

[0089] DTW method: DTW based on dynamic programming can find the optimal time alignment path between the phoneme sequence and the Mel spectrogram, ensuring that each phoneme is correctly aligned with the corresponding Mel spectrogram segment.

[0090] Attention mechanism: Modern speech synthesis systems often use an alignment model based on the attention mechanism to learn the correspondence between phonemes and Mel spectrograms through neural networks. This approach can more intelligently process phoneme sequences and speech fragments of different lengths.

[0091] The text alignment module is responsible for temporally aligning the phoneme sequence and the Mel spectrogram. The main task of this module is to generate the time interval or spectrum fragment corresponding to each phoneme, ensuring that each phoneme can be accurately mapped to the corresponding part of the speech signal when the speech is generated. The alignment information includes the time correspondence table or alignment matrix between the phoneme and the Mel spectrogram frame. The synchronization of the phoneme and the Mel spectrogram will be adjusted according to this information when the speech is generated later. The text alignment module can be trained by deep learning models such as RNN and Transformer to learn the temporal correspondence between phonemes and speech. After each phoneme is encoded, the model generates an alignment matrix by analyzing the time series of the Mel spectrogram to ensure consistency between the phoneme and the spectrum fragment. In financial applications, this helps to ensure that the speech synthesis system is consistent with the input text when generating speech, especially when synthesizing speech containing professional terms or data reports, accurate alignment is particularly important.

[0092] Example description: In the application scenario of the financial field, the text alignment module can help generate accurate voice answers. For example, in the telephone banking system, customers can enter or use voice input to ask about bank products or services. The system converts the text content entered by the customer into a phoneme sequence and extracts voice features through the Mel spectrogram. The text alignment module generates the correspondence between each phoneme and spectrum fragment through frame-level alignment, ensuring that the generated voice output is consistent with the input text in terms of time and content. This technology can ensure that voice broadcasts such as loan interest rates and financial product descriptions have high accuracy and naturalness.

[0093] By inputting the phoneme sequence and the Mel spectrogram into the text alignment module and generating alignment information, the system can ensure the precise correspondence between the phonemes and the speech signal. This makes the generated speech consistent in content and speech rhythm, especially when processing long texts or sentences with complex pronunciations, ensuring that the speech output accurately reflects the phoneme structure of the input text. In addition, this alignment method can greatly improve the accuracy and fluency of speech synthesis, especially in the financial field, such as when generating speech for compliance documents or financial reports, it can avoid the phenomenon of misalignment between phonemes and speech signals, ensuring that the speech output of professional content is accurate and clear.

[0094] S60, predicting the duration of the generated speech through a duration prediction module based on the alignment information and the potential features;

[0095] In this embodiment, the alignment information is the time correspondence between the phoneme sequence and the Mel spectrogram. It provides the start and end time information of the speech segment corresponding to each phoneme, so that the system can grasp the time mapping between phonemes and speech. The latent feature is a semantic vector generated by the text encoding module, which contains the content information of the input text. It provides high-level semantic information for speech generation by capturing the semantics and structure of the text. In the duration prediction module, the alignment information provides the precise time correspondence between each phoneme and the corresponding audio frame, while the latent feature provides the content information of the input text. These two elements together help the system predict the duration of the generated speech.

[0096] The duration prediction module is responsible for predicting the duration of the generated speech based on the input alignment information and potential features. Duration is an important factor in speech synthesis, which directly affects the naturalness and fluency of speech. Different speech contents require different speech durations, and the rhythm, pauses, and speaking speed of the speech will affect the final determination of the duration.

[0097] The alignment information provides an accurate correspondence between phonemes and Mel spectrograms, which can be used as a basic prediction indicator. The system makes a preliminary duration estimate based on the mapping time period of each phoneme in the Mel spectrogram. The latent features help the duration prediction module adjust the speech speed, pauses and other features of the speech by providing semantic information of the text to ensure that the speech generation is consistent with the text semantics. For example, a formal financial report may require a slower and rhythmic speech output, while a simple question and answer may require faster generation.

[0098] The duration prediction module is usually trained using a deep learning model. For example, based on models such as Transformer and RNN, after inputting alignment information and latent features, the model outputs the time length of the generated speech by learning the correspondence between input and duration. The training data of the model consists of a large amount of phoneme-speech alignment data with known duration. When predicting duration, the system not only considers the time markers in the alignment information, but also makes adjustments based on the text complexity, pause positions, emotional features, etc. in the latent features.

[0099] Through the duration prediction module, the system can accurately predict the duration of the generated speech based on the alignment information and potential features. The generated speech can not only keep pace with the input text, but also improve the naturalness and comprehensibility of the speech through reasonable speaking speed, pauses and rhythm. Especially when dealing with formal financial content or customer service conversations, accurate prediction of speech duration can ensure that the synthesized speech is neither too fast nor too slow, improving the user experience and speech synthesis quality.

[0100] S70, predicting the fundamental frequency of the generated speech based on the fundamental frequency information by a fundamental frequency prediction module;

[0101] In this embodiment, the fundamental frequency information is the most basic frequency information of the speech signal extracted from the Mel spectrogram by the fundamental frequency extraction module, which reflects the vibration frequency of the vocal cords. It is the main determinant of the pitch of the speech. In the process of speech generation, the fundamental frequency information is used to control the pitch, intonation and rhythmic changes of the speech. The fundamental frequency information can be used as input to train the fundamental frequency prediction module to more accurately control the fundamental frequency of the synthesized speech. The fundamental frequency information is usually represented in the form of a time series. The fundamental frequency prediction module makes further predictions based on the input fundamental frequency information to ensure that the generated speech is consistent with the reference speech in pitch and rhythm. By inputting the fundamental frequency information, the system can better capture the speaker's pitch change trend and ensure that the generated speech is natural and smooth in pitch.

[0102] The fundamental frequency prediction module uses the input fundamental frequency information, combined with other inputs such as potential features and alignment information, to predict the fundamental frequency time series of the generated speech. The purpose of this module is to generate fundamental frequency changes consistent with the input fundamental frequency information, ensuring that the pitch and intonation of the generated speech match the reference speech. The prediction module is usually trained through a deep learning model to learn the change pattern of the fundamental frequency of the speech. After receiving the fundamental frequency information extracted from the Mel spectrogram, the fundamental frequency prediction module is trained through a deep learning model (such as RNN, LSTM or Transformer) to predict the fundamental frequency at each time point. This prediction can generate smoother and more natural fundamental frequency changes, ensuring the stability of the pitch of the speech.

[0103] In addition to the input fundamental frequency information, the system also combines latent features (semantic features) to assist fundamental frequency prediction and ensure that the pitch is consistent with the semantics of the text. For example, when stating key financial terms, the system will adjust the fundamental frequency based on the latent features to generate more formal and stable pitch changes.

[0104] The fundamental frequency prediction module can be trained through a deep learning model. A common method is to model and predict the fundamental frequency time series based on a long short-term memory network (LSTM) or a recurrent neural network (RNN). During the training process, the system learns the law of generating fundamental frequencies based on a large amount of speech data with known fundamental frequencies. By inputting fundamental frequency information and potential features, the system can make high-precision predictions of the fundamental frequency of the generated speech. The predicted fundamental frequency time series can control the pitch when generating speech, ensuring that the rhythm of the generated speech is consistent with the reference speech.

[0105] Through the fundamental frequency prediction module, the system can predict the fundamental frequency time series of the generated speech based on the input fundamental frequency information, ensuring that the pitch of the synthesized speech is consistent with the reference speech. The fundamental frequency prediction module can improve the pitch stability of the speech, so that the generated speech has a natural rhythm and intonation. In scenarios where the formality and clarity of the speech need to be maintained, the predicted fundamental frequency information can effectively improve the expressiveness of the synthesized speech and ensure that the pitch changes are smooth and stable.

[0106] S80, inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to generate a reconstructed Mel spectrogram;

[0107] In this embodiment, the speech duration information generated by the duration prediction module is used to control the overall rhythm and speaking speed of the decoder-generated speech. The duration information can ensure that the generated Mel spectrogram is consistent with the input text in the time dimension. The fundamental frequency of the predicted generated speech is the fundamental frequency information generated by the fundamental frequency prediction module, which is used to control the pitch and rhythm of the decoder-generated speech. The fundamental frequency information ensures that the pitch change during speech synthesis conforms to the fundamental frequency characteristics of the reference speech. The style features extracted by the style encoding module are used to control the personalized features of the generated speech, such as intonation, rhythm, speaking speed and emotion. The style features ensure that the generated speech conforms to the style of a specific speaker, and play a key role in multi-speaker or multi-emotion speech generation, in particular.

[0108] After receiving the inputs of duration, fundamental frequency and style features, the decoder processes them through a deep learning model (such as a Transformer or Seq2Seq model) to generate a reconstructed Mel spectrogram corresponding to the input phoneme sequence and Mel spectrogram. The core task of the decoder is to fuse these different input information to generate a Mel spectrogram that accurately reflects the text content, style features and pitch changes.

[0109] The decoder is the core component of the speech synthesis system. It receives the potential features generated by the text encoding module as well as various information such as duration, fundamental frequency, style features, etc., and generates the corresponding Mel spectrogram. The Mel spectrogram is a time-frequency graph that reflects the changes in speech in time and frequency and represents the acoustic characteristics of speech.

[0110] The decoder decodes the latent features of the text into a set of Mel spectrograms by combining the input duration, fundamental frequency and style features. This process is called reconstructing the Mel spectrogram. The reconstructed Mel spectrogram reflects the acoustic characteristics of the target speech and is the basis for the subsequent generation of speech through the vocoder. The generated Mel spectrogram is consistent with the input duration and fundamental frequency, thus ensuring the naturalness and expressiveness of the synthesized speech.

[0111] The decoder is usually implemented through a deep neural network (such as RNN, Transformer), which accepts multiple inputs (such as duration, fundamental frequency, style features) and generates a mel spectrogram. During training, the decoder learns based on a large amount of known mel spectrogram data to generate accurate time-frequency graphs. In actual use, the decoder uses duration and fundamental frequency information as control factors to adjust the rhythm and pitch changes of the speech in different time periods, while the style features control the personalized performance of the generated speech.

[0112] Example description: In the financial field, the decoder can help generate synthesized speech that is consistent with the customer's voice style. For example, when a customer uses telephone banking or online customer service, the system can extract the fundamental frequency information based on the reference voice provided by the customer, and generate a response that matches the customer's tone by combining the duration and style characteristics. The decoder combines this input information with the text content to generate a reconstructed Mel spectrogram to ensure that the generated speech is both clear and personalized. In addition, when generating automatic broadcasts of financial reports, the decoder can ensure the professionalism and consistency of the voice in terms of tone and rhythm, and enhance the credibility and clarity of the voice output.

[0113] By inputting the predicted duration, fundamental frequency and style features into the decoder to generate the reconstructed Mel spectrogram, the system can accurately control the rhythm, pitch and voice style of the generated speech. Reconstructing the Mel spectrogram is a key step in generating speech. It perfectly combines the time-frequency characteristics of speech to ensure that the synthesized speech is not only consistent with the input text in content, but also natural and consistent in pitch, rhythm and expressiveness.

[0114] S90, using a vocoder to convert the reconstructed Mel spectrogram into speech audio data.

[0115] In this embodiment, the reconstructed Mel spectrogram is a time-frequency diagram generated by the decoder based on the input duration, fundamental frequency, and style features. It represents the changes in speech in time and frequency and is one of the core acoustic features of generated speech. The Mel spectrogram records the intensity changes of speech in different frequency ranges and contains information such as the pitch, tone, and rhythm of the speech. This reconstructed Mel spectrogram needs to be converted into an actual speech waveform by a vocoder to generate audible audio data.

[0116] The vocoder is a key module that converts spectral representations such as Mel spectrograms into speech waveforms. It generates audible speech by restoring the spectral information of the Mel spectrogram to a time domain signal. The main task of the vocoder is to combine the time and frequency information in the spectrum and generate a natural and smooth speech waveform.

[0117] There are several common types of vocoders used for speech synthesis:

[0118] Griffin-Lim algorithm: A traditional vocoder based on phase recovery, which converts the spectrum into a waveform by iteratively estimating the reconstructed phase. This method is simple and direct, but the quality of the generated speech is relatively low.

[0119] WaveNet Vocoder: An advanced vocoder based on deep learning that generates high-quality speech waveforms through a neural network. WaveNet is able to capture the micro-variations of speech, making the generated speech more natural.

[0120] WaveGlow: Also a neural network-based vocoder, it can quickly generate high-quality speech and is particularly suitable for real-time speech synthesis.

[0121] HiFi-GAN: A new generative adversarial network (GAN) based vocoder that can generate high-fidelity speech with a good balance between speed and quality.

[0122] The vocoder receives the reconstructed Mel spectrogram and restores it to a speech waveform in the time domain. This process consists of two steps:

[0123] Conversion from spectrum to time domain: Mel spectrogram is a speech signal represented in the frequency domain. Through the processing of the vocoder, the frequency domain signal is restored to a waveform signal in the time domain to form the actual speech data.

[0124] Phase recovery and speech reconstruction: The vocoder recovers the lost phase information and remaps the frequency and time changes into a speech waveform through complex phase estimation or deep learning methods.

[0125] Based on different vocoders, the conversion of Mel spectrogram to speech can be achieved by classical algorithms (such as Griffin-Lim) or deep learning models (such as WaveNet). Advanced deep learning vocoders such as WaveNet and HiFi-GAN can generate speech waveforms close to human voice quality and are widely used in high-demand speech synthesis scenarios.

[0126] By converting the reconstructed Mel spectrogram into speech audio data through a vocoder, the system can restore the speech representation in the frequency domain to the actual speech waveform and generate clear and natural speech audio. Using advanced deep learning vocoders (such as WaveNet or HiFi-GAN) can significantly improve the quality of generated speech, bringing it close to the level of human speech.

[0127] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a multi-module collaborative speech generation method, which obtains reference text and reference speech, converts the reference text into a phoneme sequence and generates potential features, converts the reference speech into a Mel spectrogram and extracts style features, extracts fundamental frequency information, inputs the phoneme sequence and the Mel spectrogram into a text alignment module to generate alignment information, predicts speech duration based on alignment information and potential features, predicts the fundamental frequency of speech through fundamental frequency information, inputs the predicted duration, fundamental frequency and style features into a decoder to generate a reconstructed Mel spectrogram, and finally converts the reconstructed Mel spectrogram into speech audio data through a vocoder. The present invention combines precise control of speech duration, fundamental frequency and style features, so that the generated speech can better reflect the natural rhythm and pitch changes of the text content, and enhances the rhythm and expressiveness of the speech, thereby improving the naturalness and fidelity of speech synthesis.

[0128] In one embodiment, before the above S20, the method further includes:

[0129] S101, obtaining a training sample, wherein the training sample includes an initial phoneme sequence of a text sample and an initial Mel spectrogram of a speech sample;

[0130] S102, inputting the initial phoneme sequence into a text encoding module to generate training latent features, analyzing the loss value between the training latent features and the target latent features of the training samples through a loss function, and updating the parameters of the text encoding module according to the loss value;

[0131] S103, inputting the initial Mel spectrogram into a style encoding module to generate a training style feature vector, analyzing the loss value between the training style feature and the target style feature of the training sample through a loss function, and adjusting the parameters of the style encoding module according to the loss value;

[0132] S104, inputting the initial Mel spectrogram into a fundamental frequency extraction module to generate training fundamental frequency information, analyzing the loss value between the training fundamental frequency information and the target fundamental frequency information of the training sample through a loss function, and adjusting the parameters of the fundamental frequency extraction module according to the loss value;

[0133] S105, inputting the initial phoneme sequence and the initial Mel spectrogram into a text alignment module to generate training alignment information, and using the training alignment information to perform time alignment on the initial phoneme sequence and the initial Mel spectrogram, and updating the parameters of the text alignment module through error feedback information;

[0134] S106, inputting the training potential features, training style feature vectors, training fundamental frequency information and training alignment information into a decoder to generate a training reconstructed Mel spectrogram;

[0135] S107, comparing the training reconstructed Mel spectrogram with the target Mel spectrogram of the speech sample through a discriminator, and updating the parameters of the decoder through comparison feedback information of the discriminator.

[0136] In this embodiment, the training sample includes the initial phoneme sequence of the text sample and the initial Mel spectrogram of the speech sample. This is the basic data required for system training. The initial phoneme sequence is the pronunciation symbol sequence obtained after the text is converted to phonemes, and the initial Mel spectrogram is the spectrum diagram representation of the reference speech, which is used to train various modules in the system. The text sample is converted to a phoneme sequence through a phoneme converter, and the speech sample is converted to a Mel spectrogram through a short-time Fourier transform (STFT). After obtaining the training sample, the system uses these two samples as input and uses them for the training of subsequent modules respectively.

[0137] The text encoding module is responsible for converting the initial phoneme sequence into latent features. Latent features are the semantic representation of text in speech generation and contain the content information of speech generation. The system encodes the phoneme sequence into latent features through deep learning models such as LSTM or Transformer. The system inputs the phoneme sequence into the trained text encoding model, which analyzes the phoneme sequence frame by frame through a multi-layer neural network and extracts its high-level semantic features. The generated latent features are used to guide subsequent speech generation.

[0138] The system uses a loss function to evaluate the difference between the training latent features and the target latent features. The target latent features are pre-trained model outputs that are used to compare the newly generated features. The loss value reflects the closeness of the features generated by the system to the target features. The system adjusts the model parameters through back propagation to make its output closer to the target features. Using mean square error (MSE) or cross entropy as the loss function, the system updates the weights of the text encoding module through the gradient descent algorithm so that the generated latent features better capture the semantic information of the text.

[0139] The style encoding module extracts style features from the Mel spectrogram. Style features include timbre, emotion, intonation, etc. The style feature vector is used to guide the generation of speech with personalized features. The initial Mel spectrogram contains the basic acoustic information of the speech, and the style features are extracted through the deep learning model. The Mel spectrogram is input into the deep learning model (such as CNN) to generate training style feature vectors. These vectors represent non-linguistic features such as emotion and intonation of the speech, which are used to control the expressiveness of the speech in the subsequent generation stage.

[0140] The system compares the generated style features with the target style features, calculates the loss, and adjusts the parameters of the style encoding module based on the difference. The target style features are usually standard features obtained through large-scale data training, which are used to guide the adjustment of the model. The difference between style features is quantified using a loss function (such as L2 loss), and the model parameters are updated through back propagation so that the features generated by the style encoding module are more consistent with the training objectives.

[0141] The fundamental frequency extraction module extracts fundamental frequency information from the Mel spectrogram. The fundamental frequency is the main pitch information of speech and reflects the pitch changes in speech. The training fundamental frequency information is used to guide the pitch control in the generation stage. The fundamental frequency information in the Mel spectrogram is extracted by traditional fundamental frequency extraction algorithms (such as the YIN algorithm) or deep learning models. The fundamental frequency information is usually stored in the form of a time series, indicating the change of pitch over time.

[0142] The system compares the generated training fundamental frequency information with the target fundamental frequency information, calculates the fundamental frequency difference, and adjusts the parameters of the fundamental frequency extraction module to make the generated fundamental frequency closer to the fundamental frequency characteristics of the target sample. Use a loss function based on time series difference, such as the DTW (Dynamic Time Warping) algorithm, calculate the difference between the generated fundamental frequency information and the target fundamental frequency, and update the model parameters through the back propagation algorithm.

[0143] The text alignment module generates training alignment information by aligning the phoneme sequence with the Mel spectrogram. The alignment information is the corresponding time period of each phoneme in the Mel spectrogram, which ensures that the generated speech is time synchronized with the text content. Through dynamic time warping (DTW) or an alignment model based on the attention mechanism, the time mapping between the phoneme and the Mel spectrogram frame is generated and the training alignment information is output.

[0144] The system ensures the time synchronization between the initial phoneme sequence and the Mel spectrogram by training alignment information, and adjusts the parameters of the alignment model through error feedback to make the alignment between the phoneme and the Mel spectrogram more accurate. The system calculates the alignment accuracy through the alignment error, and adjusts the parameters of the text alignment module through back propagation to gradually optimize the alignment effect.

[0145] The decoder receives these feature information and generates a training reconstructed Mel spectrogram. The decoder reconstructs a high-quality Mel spectrogram as the basis for generating speech by combining latent features, style features, fundamental frequency and alignment information. The decoder is implemented based on a deep learning model (such as Transformer, LSTM), and after inputting multiple feature information, it generates a reconstructed Mel spectrogram corresponding to the target speech.

[0146] The discriminator is used to judge the difference between the generated training reconstructed Mel spectrogram and the target Mel spectrogram, and adjust the parameters of the decoder through a feedback mechanism. The discriminator is similar to the discriminator in GAN. By comparing the real and generated Mel spectrograms, the quality of the generated Mel spectrogram is gradually improved. The generated Mel spectrogram is evaluated by the discriminator model of the generative adversarial network (GAN). The closer the generated Mel spectrogram is to the target Mel spectrogram, the more the output of the decoder meets the requirements. The feedback of the discriminator is used to further optimize the decoder.

[0147] In this embodiment, the system can gradually improve the quality of generated speech by inputting the initial phoneme sequence and Mel spectrogram of the training sample into each module and optimizing them. The reconstructed Mel spectrogram finally generated is highly close to the target speech, ensuring that the generated speech is consistent with the reference speech in terms of pitch, rhythm, duration, etc. This training process enables the model to effectively handle complex speech generation tasks, especially with significant improvements in the personalization, accuracy and naturalness of speech generation.

[0148] In one embodiment, after the above S207, the method further includes:

[0149] S108, fixing the parameters of the text encoding module, style encoding module, baseband extraction module, text alignment module and decoder;

[0150] S109, using the training alignment information, predicting the speech duration generated by the training through a duration prediction module, and adjusting the parameters of the duration prediction module according to the speech duration generated by the training;

[0151] S110, using the training style features and the training potential features, predicting the fundamental frequency information generated by the training through a fundamental frequency prediction module, and adjusting the parameters of the fundamental frequency prediction module according to the fundamental frequency information generated by the training;

[0152] S111, adjusting the speech time distribution in the generation phase based on the speech duration information generated by the training, and adjusting the speech pitch in the generation phase based on the fundamental frequency information generated by the training;

[0153] S112, optimizing parameters of the duration prediction module and the fundamental frequency prediction module based on the difference between the duration information generated by training and the duration of the target sample, and the difference between the fundamental frequency information generated by training and the fundamental frequency of the target sample.

[0154] In this embodiment, the discriminator is responsible for comparing the generated training reconstructed Mel spectrogram with the target Mel spectrogram of the real speech sample, adjusting the decoder parameters through the feedback mechanism, and gradually optimizing the quality of the generated Mel spectrogram. The discriminator can be regarded as a quality control mechanism, similar to the discriminator in the generative adversarial network, which is used to improve the authenticity of the generated results. The discriminator based on the generative adversarial network (GAN) evaluates the generated Mel spectrogram, outputs feedback to the decoder model, updates its parameters, and makes the Mel spectrogram generated by the decoder closer to the real speech.

[0155] After the first phase of training is completed, the parameters of the text encoding module, style encoding module, base frequency extraction module, text alignment module, and decoder have been optimized to a certain extent. At this time, it is necessary to fix the parameters of these modules to ensure that they are no longer updated, so that the optimization of the duration prediction and base frequency prediction modules can be focused on in the subsequent training. By freezing the weights and parameters of the model, the optimization results of these modules during the training phase will not change. The subsequent training will only affect the duration prediction module and the base frequency prediction module.

[0156] The duration prediction module predicts the duration of each speech segment based on the generated training alignment information. The alignment information contains the time mapping of the phoneme sequence and the Mel spectrogram, so it can effectively guide the prediction of speech duration. The duration prediction module updates its own parameters based on the difference in duration between the actual speech sample and the generated speech. The duration prediction module uses the alignment information as input, predicts the duration of speech through a time series model such as a recurrent neural network (RNN) or LSTM, and optimizes the model parameters by minimizing the duration error (such as the MSE loss function).

[0157] The fundamental frequency prediction module predicts the fundamental frequency information of the generated speech based on the training style features and the training latent features. The style features control the rhythm and personalized expression of the speech, while the latent features control the content information of the speech. The fundamental frequency prediction module adjusts the parameters according to the difference between the fundamental frequency information generated by the training and the target fundamental frequency information. The fundamental frequency prediction module predicts the fundamental frequency time series by combining the style features and the latent features through a deep neural network. During the training process, the model adjusts the parameters according to the difference between the fundamental frequency of the generated speech and the fundamental frequency of the target sample to ensure accurate fundamental frequency prediction.

[0158] In the speech generation stage, the system uses the speech duration information and fundamental frequency information generated during training to control the time distribution and pitch variation of the final generated speech. This ensures that the generated speech has a natural rhythm and pitch, and conforms to the style and intonation of the reference speech. The decoder receives the duration information to adjust the time axis distribution of the speech to ensure the smooth rhythm of the generated speech; the fundamental frequency information is used to adjust the pitch to make the speech tone natural. This stage makes final adjustments based on the outputs of the duration prediction module and the fundamental frequency prediction module.

[0159] The system further optimizes the parameters of the duration prediction and fundamental frequency prediction modules by comparing the duration and fundamental frequency information generated by training with the target sample. This step is to optimize the model through difference feedback to ensure that the generated speech is highly consistent with the target sample in duration and pitch. By calculating the difference in duration and fundamental frequency between the generated speech and the target speech, the system uses a loss function (such as mean square error MSE) to calculate the error, and optimizes the parameters of the duration prediction module and fundamental frequency prediction module through back propagation to improve the prediction accuracy of the model.

[0160] In this embodiment, through the dual feedback mechanism of the discriminator and the prediction module, the system can gradually improve the duration and fundamental frequency prediction accuracy of the generated speech. By fixing the parameters of the previous modules and centrally optimizing the duration and fundamental frequency prediction modules, the system can ensure that the rhythm and pitch of the generated speech are natural and stable, further improving the expressiveness and auditory quality of the speech. This feedback optimization-based mechanism is particularly suitable for scenarios that require highly accurate speech output, such as automatic generation of financial reports and high-precision speech synthesis in voice assistants.

[0161] In one embodiment, the above S20 includes:

[0162] S201, performing a preprocessing operation on the reference text, wherein the preprocessing operation includes removing non-language symbols in the reference text;

[0163] S202, segmenting the reference text after the preprocessing operation by a word segmenter, identifying and extracting multiple words in the reference text;

[0164] S203, converting the plurality of words into corresponding phoneme sequences through a phoneme converter;

[0165] S204, inputting the phoneme sequence into a text encoding module, and the text encoding module extracts features from the phoneme sequence based on a trained neural network model to generate potential features.

[0166] In this embodiment, the preprocessing operation is used to clean up non-language symbols in the reference text, including punctuation marks, emoticons, numbers, etc. Non-language symbols have no practical significance for speech generation, so they need to be removed before phoneme conversion to ensure that the generated phoneme sequence contains only meaningful pronunciation units. The system scans the text through regular expressions or other text processing tools to automatically remove non-language symbols. The cleaned text is more concise and suitable for subsequent phoneme conversion and feature extraction.

[0167] The word segmenter is used to break down the reference text into vocabulary units, which are the basis of the speech content. In some languages, such as Chinese, word segmentation is to divide the continuous text into more easily processable speech fragments. In other languages, such as English, word segmentation is to identify word boundaries. The word segmenter uses a dictionary-based or statistical model-based method to segment words. It scans the text, identifies individual words, and uses these words as input for subsequent processing. Implementation methods may include statistical-based word segmentation methods, maximum matching methods, or deep learning models (such as BERT) to automatically segment words.

[0168] The phoneme converter is used to convert the segmented vocabulary into a phoneme sequence. Phoneme is the smallest pronunciation unit for speech generation, and each phoneme corresponds to a specific pronunciation. In the speech generation system, the phoneme sequence is the basis for converting text to speech. The phoneme converter uses a phonetic dictionary or a rule-based model to convert each word into a corresponding phoneme sequence. For multilingual speech generation systems, the phoneme converter needs to accurately map vocabulary to phonemes according to the pronunciation rules of different languages.

[0169] The text encoding module is used to encode the phoneme sequence and generate latent features. Latent features are high-level semantic representations of text that are used to guide the speech generation process. This encoding process is usually completed by a neural network model (such as LSTM, GRU, or Transformer). The system inputs the phoneme sequence into a pre-trained neural network model, which analyzes the phoneme sequence frame by frame and extracts feature information from it. The generated latent features contain the text semantic information required for speech generation, as well as the pronunciation order of the text in the time dimension.

[0170] In this embodiment, the system can effectively generate latent features from the text by preprocessing, word segmentation, phoneme conversion and text encoding of the reference text. These latent features are the core of speech generation, containing the semantic information and pronunciation order of the text, ensuring that the generated speech is consistent with the input text. By using the trained neural network model for encoding, the system can better capture the implicit semantic relationship in the text, thereby generating natural and fluent speech, which is suitable for complex speech synthesis scenarios.

[0171] In one embodiment, the above S30 includes:

[0172] S301, performing a preprocessing operation on the reference speech, wherein the processing operation includes a denoising operation and a volume normalization operation;

[0173] S302, performing a pre-emphasis filtering operation on the reference speech after the pre-processing operation;

[0174] S303, inputting the pre-emphasized filtered reference speech into a short-time Fourier transform module to obtain a frequency spectrum representation of the reference speech;

[0175] S304, converting the frequency spectrum representation into a Mel spectrum through a Mel filter bank to generate a Mel spectrogram;

[0176] S305: Input the Mel spectrum into a style encoding module. The style encoding module analyzes the Mel spectrum using deep learning and extracts style features from the Mel spectrum.

[0177] In this embodiment, the reference speech needs to be preprocessed before being input into the model to improve the accuracy of subsequent processing. The denoising operation is used to eliminate background noise and ensure the clarity of the input signal; volume normalization is used to unify the volume level of the speech signal so that the model can process speech samples of different volumes and avoid errors caused by volume differences. Denoising and volume normalization operations are implemented through common audio processing tools or algorithms (such as filters, low-pass filters, amplitude normalization, etc.). The denoising algorithm can use a frequency domain denoising method, or an adaptive denoising model based on a neural network. Volume normalization is completed by adjusting the signal amplitude to a preset range.

[0178] Pre-emphasis filtering is used to enhance the high-frequency components in speech signals, especially when some high-frequency features are easily overlooked. Pre-emphasis can help improve speech clarity and better reflect speech details in the spectrum. Pre-emphasis is a common filtering operation, usually implemented using a difference equation, which enhances high-frequency components and improves the effect of subsequent spectrum analysis.

[0179] Short-time Fourier transform (STFT) is used to convert speech signals from the time domain to the frequency domain. Through STFT, the speech signal is decomposed into multiple frequency components to capture the changes in the signal in the time and frequency dimensions. The spectral representation output by STFT is the basis for the generation of Mel spectrogram. STFT divides the time domain signal into small segments by processing the speech signal with a window function, and performs Fourier transform on each segment. In the implementation process, Hamming window or other window functions are often used to segment the signal, and Fourier transform is applied to each segment to generate a spectrum.

[0180] The Mel filter bank is a set of specially designed filters that converts the STFT spectrum into a Mel frequency domain representation based on the sensitivity of the human ear to different frequencies. The Mel spectrum is more in line with human hearing characteristics and can better reflect the perceptual characteristics of speech signals. The spectrum generated by the STFT is filtered through a set of Mel filters, and the bandwidth and center frequency of the filter gradually become wider as the frequency increases. The Mel filter bank generates a Mel spectrum by summing the amplitude of the STFT spectrum. The final result is a low-dimensional Mel spectrogram that can preserve the main acoustic features of speech.

[0181] The style encoding module analyzes the Mel spectrogram through a deep learning model to extract the style features of the speech. Style features reflect the speaker's tone, emotion, timbre and other information, and are often used for multi-style speech generation or speaker voice conversion. Using a convolutional neural network (CNN) or other deep learning models, the style encoding module analyzes the Mel spectrogram and extracts high-dimensional features from it. These features represent the speaker's style and emotion, and are usually encoded into a low-dimensional style vector as input for the subsequent generation stage.

[0182] In this embodiment, the system can generate high-quality Mel spectrograms through preprocessing, pre-emphasis, STFT conversion and Mel filtering. These Mel spectrograms can accurately reflect the acoustic features in the speech signal, providing a reliable basis for subsequent style coding. The style features extracted by the style coding module through the deep learning model can be used to generate speech with personalized style, so that the system can generate more natural and diverse speech synthesis results, especially suitable for scenes requiring multi-style speech or emotional speech.

[0183] In one embodiment, the above S50 includes:

[0184] S501, inputting the phoneme sequence and Mel spectrogram into a text alignment module for alignment processing;

[0185] S502, performing frame-level alignment on the phoneme sequence and the Mel spectrogram by the text alignment module;

[0186] S503, generating a time correspondence table between each phoneme and the corresponding audio frame according to the frame-level alignment result;

[0187] S504, determining the alignment position of each phoneme in the Mel spectrogram based on the time correspondence table;

[0188] S505, integrating the alignment positions of all phonemes in the Mel spectrogram to generate complete alignment information.

[0189] In this embodiment, the phoneme sequence and the Mel spectrogram represent the pronunciation information of the text and the spectrum information of the speech, respectively. The purpose of the alignment process is to match the phoneme sequence and the Mel spectrogram on the time axis so that each phoneme can correspond to a certain period in the Mel spectrogram. By taking the phoneme sequence and the Mel spectrogram as input, the text alignment module starts frame-level processing, and uses dynamic time warping (DTW) or an alignment algorithm based on an attention mechanism to process phoneme sequences and Mel spectrograms of different lengths to ensure that each phoneme finds its corresponding area in the Mel spectrogram.

[0190] Frame-level alignment means accurately matching each phoneme in the phoneme sequence with a frame in the Mel spectrogram to ensure the time synchronization of the phoneme and the speech frame. Each frame corresponds to a specific time segment, so that the system can capture the time position of the phoneme in the Mel spectrogram. The text alignment module uses frame-level dynamic time warping (frame-level DTW) or a deep learning-based attention model (such as a Transformer-based alignment model) to align the phoneme sequence with the Mel spectrogram. Frame-level alignment adjusts the temporal relationship between phonemes and frames by minimizing the time error.

[0191] The result of the alignment is a time correspondence table that lists the audio frames corresponding to each phoneme. The time correspondence table describes in detail the start and end time periods of each phoneme to ensure that the phoneme accurately matches the frame segments in the Mel spectrogram. After frame-level alignment, the system generates a time correspondence table that records the position of each phoneme in the Mel spectrogram. This correspondence table is usually stored in the form of a timestamp, indicating the specific time relationship between the phoneme and its corresponding frame.

[0192] The system uses the time correspondence table to determine the position of each phoneme in the Mel spectrogram. This includes the start and end frames of each phoneme in the Mel spectrogram. Through this process, the specific position of the phoneme in the speech spectrum can be determined. By scanning the time correspondence table, the system can extract the time period information of each phoneme in the Mel spectrogram. The system matches each phoneme with its corresponding spectrum frame one by one to ensure accurate alignment between the phoneme sequence and the Mel spectrogram.

[0193] After integrating the alignment position of each phoneme, the system generates a complete alignment information, which includes the time position of all phonemes in the Mel spectrogram. This alignment information is an important input for the subsequent speech generation stage, which ensures that the generated speech and text content are consistent in time. The system generates the final alignment information by summarizing the start and end frames of each phoneme. This information is stored in the form of a list or time table as input data for the subsequent decoder or speech generation model to ensure that the generated speech is synchronized with the text.

[0194] In this embodiment, the system can accurately determine the time correspondence between phonemes and Mel spectrograms through the frame-level alignment achieved by the text alignment module. This alignment information provides accurate timeline information for subsequent speech generation, ensuring that the generated speech is completely synchronized with the text content. Through dynamic time warping or alignment methods based on attention mechanisms, the system can handle inputs of different lengths, ensuring that the alignment process is efficient and accurate, and improving the naturalness and accuracy of speech generation.

[0195] In one embodiment, the above S80 includes:

[0196] S801, inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to control the generation process of the Mel spectrogram;

[0197] S802, adjusting the rhythmic features and timbre features in the Mel spectrogram generated by the decoder based on the style features;

[0198] S803, adjusting the temporal characteristics of the Mel spectrogram generated by the decoder based on the predicted speech duration;

[0199] S804, adjusting the pitch feature of the Mel spectrogram generated by the decoder based on the predicted fundamental frequency;

[0200] S805, generating a final adjusted reconstructed Mel spectrum.

[0201] In this embodiment, the decoder receives the predicted speech duration, fundamental frequency and style features as input. These inputs are used to control the generation process of the Mel spectrogram, where the duration determines the time axis distribution of the speech, the fundamental frequency determines the pitch of the speech, and the style features affect the rhythm, emotion and timbre of the speech. By inputting the predicted duration, fundamental frequency and style features into the decoder module, the decoder uses this information to guide the generation of the Mel spectrogram. The decoder uses a deep learning model (such as a generative model based on LSTM, Transformer) to fuse these input features and generate a Mel spectrogram that meets the feature requirements frame by frame.

[0202] Style features are used to adjust the rhythm (rhythm and intonation) and timbre (sound quality) of the generated speech. Rhythmic features include changes in speech rate and intonation, while timbre features determine the speaker's voice style, such as heavy, soft, etc. The decoder controls these aspects of the generated Mel spectrogram through style features, so that the generated speech has a personalized style. When generating the Mel spectrogram, the decoder adjusts the rhythm and timbre of the speech through style features. The specific implementation can be achieved by introducing a style control module (such as the AdaIN style transformation method) to adjust the parameters of the decoder so that the generated Mel spectrogram meets the specified style requirements.

[0203] Speech duration prediction is used to control the timing features of the generated speech to ensure that the playback duration of the speech matches the actual predicted time distribution. The timing features refer to the distribution of each phoneme or speech segment on the time axis. The decoder adjusts the timing features to ensure that the temporal rhythm of the generated speech is natural and in line with expectations. The decoder uses the output of the duration prediction module to control the duration of each phoneme in the Mel spectrogram and ensure that the distribution of the generated Mel spectrogram on the time axis matches the actual predicted duration. Common methods include adjusting the timing features through RNN or attention mechanisms to make the speech duration consistent with the target duration.

[0204] The fundamental frequency (F0) determines the pitch variation of speech. Based on the predicted fundamental frequency information, the pitch features in the Mel spectrogram generated by the decoder are adjusted so that the pitch of the generated speech is consistent with the actual predicted fundamental frequency. Pitch features usually include the pitch variation of the speaker's voice. The decoder adjusts the pitch of the Mel spectrogram based on the output of the fundamental frequency prediction module. Common techniques include adjusting the pitch information of the Mel spectrogram to match the predicted fundamental frequency through a time series model or a self-attention mechanism. The pitch of the generated Mel spectrogram at different time periods will be consistent with the input fundamental frequency.

[0205] After adjusting the rhythm, timbre, timing and pitch, the decoder generates the final reconstructed Mel spectrogram. This reconstructed Mel spectrogram is the final output of speech generation, which contains the predicted speech features, ensuring that the generated speech is consistent with the input data in terms of emotional expression, timbre, rhythm and pitch. The decoder uses the duration, fundamental frequency and style features of all inputs to generate the adjusted Mel spectrogram. The generated Mel spectrogram contains a complete sequence of audio frames, which is used for further processing by the speech synthesizer to finally generate speech audio.

[0206] In this embodiment, the system can generate a highly customized Mel spectrogram by inputting the predicted speech duration, fundamental frequency and style features into the decoder. The decoder makes the generated speech more natural and personalized by accurately controlling the pitch, timing, rhythm and timbre, meeting the user's expectations. It can ensure that the generated speech is not only consistent with the target in terms of time distribution, but also can achieve stylized speech that matches the expected emotion and timbre, which is suitable for use in scenarios such as intelligent voice assistants and personalized voice broadcasts.

[0207] In one embodiment, a multi-module collaborative speech generation device is provided, and the multi-module collaborative speech generation device corresponds one-to-one to the multi-module collaborative speech generation method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a multi-module collaborative speech generation device of the present invention. Data acquisition module 10, text encoding module 20, style encoding module 30, fundamental frequency extraction module 40, text alignment module 50, duration prediction module 60, fundamental frequency prediction module 70, decoder module 80 and vocoder module 90. The functional modules are described in detail as follows:

[0208] A data acquisition module 10 is used to acquire reference text and reference speech;

[0209] A text encoding module 20, used for converting the reference text into a phoneme sequence, and inputting the phoneme sequence into a text encoding module to generate latent features;

[0210] A style encoding module 30, configured to convert the reference speech into a Mel spectrogram, and input the Mel spectrogram into a style encoding module, and the style encoding module extracts style features;

[0211] A fundamental frequency extraction module 40, configured to extract fundamental frequency information from the Mel spectrogram through a fundamental frequency extraction module;

[0212] A text alignment module 50, used for inputting the phoneme sequence and the Mel spectrogram into a text alignment module, and the text alignment module generates alignment information;

[0213] A duration prediction module 60, configured to predict the duration of the generated speech through a duration prediction module based on the alignment information and the latent features;

[0214] A fundamental frequency prediction module 70, configured to predict the fundamental frequency of the generated speech based on the fundamental frequency information through a fundamental frequency prediction module;

[0215] A decoder module 80, used for inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to generate a reconstructed Mel spectrogram;

[0216] The vocoder module 90 is used to convert the reconstructed Mel spectrogram into speech audio data using a vocoder.

[0217] In one embodiment, the text encoding module 20 is specifically used for:

[0218] Acquire a training sample, wherein the training sample includes an initial phoneme sequence of a text sample and an initial Mel spectrogram of a speech sample;

[0219] Inputting the initial phoneme sequence into a text encoding module to generate training latent features, analyzing the loss value between the training latent features and the target latent features of the training sample through a loss function, and updating the parameters of the text encoding module according to the loss value;

[0220] Input the initial Mel spectrogram into a style encoding module to generate a training style feature vector, analyze the loss value between the training style feature and the target style feature of the training sample through a loss function, and adjust the parameters of the style encoding module according to the loss value;

[0221] Input the initial Mel spectrogram into a fundamental frequency extraction module to generate training fundamental frequency information, analyze the loss value between the training fundamental frequency information and the target fundamental frequency information of the training sample through a loss function, and adjust the parameters of the fundamental frequency extraction module according to the loss value;

[0222] Inputting the initial phoneme sequence and the initial Mel spectrogram into a text alignment module to generate training alignment information, and using the training alignment information to perform time alignment on the initial phoneme sequence and the initial Mel spectrogram, and updating the parameters of the text alignment module through error feedback information;

[0223] Inputting the training potential features, training style feature vectors, training fundamental frequency information and training alignment information into a decoder to generate a training reconstructed Mel spectrogram;

[0224] The training reconstructed Mel spectrogram is compared with the target Mel spectrogram of the speech sample through a discriminator, and the parameters of the decoder are updated through the comparison feedback information of the discriminator.

[0225] In one embodiment, the text encoding module 20 is specifically used for:

[0226] Fixing parameters of the text encoding module, style encoding module, baseband extraction module, text alignment module and decoder;

[0227] Using the training alignment information, predicting the speech duration generated by the training through a duration prediction module, and adjusting the parameters of the duration prediction module according to the speech duration generated by the training;

[0228] Using the training style features and the training potential features, predicting the fundamental frequency information generated by the training through a fundamental frequency prediction module, and adjusting the parameters of the fundamental frequency prediction module according to the fundamental frequency information generated by the training;

[0229] Adjusting the speech time distribution in the generation phase based on the speech duration information generated by the training, and adjusting the speech pitch in the generation phase based on the fundamental frequency information generated by the training;

[0230] Based on the difference between the duration information generated by training and the duration of the target sample, and the difference between the fundamental frequency information generated by training and the fundamental frequency of the target sample, the parameters of the duration prediction module and the fundamental frequency prediction module are optimized.

[0231] In one embodiment, the text encoding module 20 is specifically used for:

[0232] Performing a preprocessing operation on the reference text, wherein the preprocessing operation includes removing non-language symbols in the reference text;

[0233] The reference text after the preprocessing operation is segmented by a word segmenter to identify and extract multiple words in the reference text;

[0234] Convert the plurality of words into corresponding phoneme sequences through a phoneme converter;

[0235] The phoneme sequence is input into a text encoding module, and the text encoding module extracts features from the phoneme sequence based on a trained neural network model to generate potential features.

[0236] In one embodiment, the style encoding module 30 is specifically configured to:

[0237] Performing a preprocessing operation on the reference speech, wherein the processing operation includes a denoising operation and a volume normalization operation;

[0238] performing a pre-emphasis filtering operation on the reference speech after the pre-processing operation;

[0239] Inputting the pre-emphasized filtered reference speech into a short-time Fourier transform module to obtain a spectrum representation of the reference speech;

[0240] The frequency spectrum representation is converted into a Mel spectrum by a Mel filter bank to generate a Mel spectrogram;

[0241] The Mel spectrum is input into a style encoding module, and the style encoding module analyzes the Mel spectrum using deep learning and extracts style features from the Mel spectrum.

[0242] In one embodiment, the text alignment module 50 is specifically configured to:

[0243] Input the phoneme sequence and Mel spectrogram into a text alignment module for alignment processing;

[0244] Performing frame-level alignment on the phoneme sequence and the Mel spectrogram by using the text alignment module;

[0245] According to the frame-level alignment result, a time correspondence table between each phoneme and the corresponding audio frame is generated;

[0246] Based on the time correspondence table, determining the alignment position of each phoneme in the Mel spectrogram;

[0247] The alignment positions of all phonemes in the Mel spectrogram are integrated to generate complete alignment information.

[0248] In one embodiment, the decoder module 80 is specifically configured to:

[0249] Inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to control the generation process of the Mel spectrogram;

[0250] Adjusting the rhythmic features and timbre features in the Mel spectrogram generated by the decoder based on the style features;

[0251] Adjust the temporal features of the Mel spectrogram generated by the decoder based on the predicted speech duration;

[0252] Adjust the pitch characteristics of the Mel spectrogram generated by the decoder based on the predicted fundamental frequency;

[0253] Generate the final adjusted reconstructed Mel spectrum.

[0254] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a multi-module collaborative speech generation method are implemented.

[0255] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the user side of a multi-module collaborative speech generation method are implemented.

[0256] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0257] Obtain reference text and reference speech;

[0258] Converting the reference text into a phoneme sequence, and inputting the phoneme sequence into a text encoding module to generate latent features;

[0259] Convert the reference speech into a Mel spectrogram, input the Mel spectrogram into a style encoding module, and extract style features by the style encoding module;

[0260] Extracting fundamental frequency information from the Mel spectrogram by a fundamental frequency extraction module;

[0261] Input the phoneme sequence and the Mel spectrogram into a text alignment module, and the text alignment module generates alignment information;

[0262] Based on the alignment information and the latent features, predicting the duration of the generated speech by a duration prediction module;

[0263] Predicting the fundamental frequency of the generated speech based on the fundamental frequency information by a fundamental frequency prediction module;

[0264] Inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to generate a reconstructed Mel spectrogram;

[0265] The reconstructed Mel spectrogram is converted into speech audio data using a vocoder.

[0266] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0267] Obtain reference text and reference speech;

[0268] Converting the reference text into a phoneme sequence, and inputting the phoneme sequence into a text encoding module to generate latent features;

[0269] Convert the reference speech into a Mel spectrogram, input the Mel spectrogram into a style encoding module, and extract style features by the style encoding module;

[0270] Extracting fundamental frequency information from the Mel spectrogram by a fundamental frequency extraction module;

[0271] Input the phoneme sequence and the Mel spectrogram into a text alignment module, and the text alignment module generates alignment information;

[0272] Based on the alignment information and the latent features, predicting the duration of the generated speech by a duration prediction module;

[0273] Predicting the fundamental frequency of the generated speech based on the fundamental frequency information by a fundamental frequency prediction module;

[0274] Inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to generate a reconstructed Mel spectrogram;

[0275] The reconstructed Mel spectrogram is converted into speech audio data using a vocoder.

[0276] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0277] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0278] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0279] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A multi-module collaborative speech generation method, characterized in that: The following steps are involved: Obtain reference text and reference speech; Converting the reference text into a phoneme sequence, and inputting the phoneme sequence into a text encoding module to generate latent features; Convert the reference speech into a Mel spectrogram, input the Mel spectrogram into a style encoding module, and extract style features by the style encoding module; Extracting fundamental frequency information from the Mel spectrogram by a fundamental frequency extraction module; Input the phoneme sequence and the Mel spectrogram into a text alignment module, and the text alignment module generates alignment information; Based on the alignment information and the latent features, predicting the duration of the generated speech by a duration prediction module; Predicting the fundamental frequency of the generated speech based on the fundamental frequency information by a fundamental frequency prediction module; Inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to generate a reconstructed Mel spectrogram; The reconstructed Mel spectrogram is converted into speech audio data using a vocoder.

2. The multi-module collaborative speech generation method according to claim 1, characterized in that: The reference text is converted into a phoneme sequence, and the phoneme sequence is input into a text encoding module, before generating potential features, the method further includes: Acquire a training sample, wherein the training sample includes an initial phoneme sequence of a text sample and an initial Mel spectrogram of a speech sample; Inputting the initial phoneme sequence into a text encoding module to generate training latent features, analyzing the loss value between the training latent features and the target latent features of the training sample through a loss function, and updating the parameters of the text encoding module according to the loss value; Input the initial Mel spectrogram into a style encoding module to generate a training style feature vector, analyze the loss value between the training style feature and the target style feature of the training sample through a loss function, and adjust the parameters of the style encoding module according to the loss value; Input the initial Mel spectrogram into a fundamental frequency extraction module to generate training fundamental frequency information, analyze the loss value between the training fundamental frequency information and the target fundamental frequency information of the training sample through a loss function, and adjust the parameters of the fundamental frequency extraction module according to the loss value; Inputting the initial phoneme sequence and the initial Mel spectrogram into a text alignment module to generate training alignment information, and using the training alignment information to perform time alignment on the initial phoneme sequence and the initial Mel spectrogram, and updating the parameters of the text alignment module through error feedback information; Inputting the training potential features, training style feature vectors, training fundamental frequency information and training alignment information into a decoder to generate a training reconstructed Mel spectrogram; The training reconstructed Mel spectrogram is compared with the target Mel spectrogram of the speech sample through a discriminator, and the parameters of the decoder are updated through the comparison feedback information of the discriminator.

3. The multi-module collaborative speech generation method according to claim 2, characterized in that: After comparing the training reconstructed Mel spectrogram with the target Mel spectrogram of the speech sample through the discriminator and updating the parameters of the decoder through the comparison feedback information of the discriminator, the method further includes: Fixing parameters of the text encoding module, style encoding module, baseband extraction module, text alignment module and decoder; Using the training alignment information, predicting the speech duration generated by the training through a duration prediction module, and adjusting the parameters of the duration prediction module according to the speech duration generated by the training; Using the training style features and the training potential features, predicting the fundamental frequency information generated by the training through a fundamental frequency prediction module, and adjusting the parameters of the fundamental frequency prediction module according to the fundamental frequency information generated by the training; Adjusting the speech time distribution in the generation phase based on the speech duration information generated by the training, and adjusting the speech pitch in the generation phase based on the fundamental frequency information generated by the training; Based on the difference between the duration information generated by training and the duration of the target sample, and the difference between the fundamental frequency information generated by training and the fundamental frequency of the target sample, the parameters of the duration prediction module and the fundamental frequency prediction module are optimized.

4. The multi-module collaborative speech generation method according to claim 1, characterized in that: The reference text is converted into a phoneme sequence, and the phoneme sequence is input into a text encoding module to generate potential features, including: Performing a preprocessing operation on the reference text, wherein the preprocessing operation includes removing non-language symbols in the reference text; The reference text after the preprocessing operation is segmented by a word segmenter to identify and extract multiple words in the reference text; Convert the plurality of words into corresponding phoneme sequences through a phoneme converter; The phoneme sequence is input into a text encoding module, and the text encoding module extracts features from the phoneme sequence based on a trained neural network model to generate potential features.

5. The multi-module collaborative speech generation method as claimed in claim 1, characterized in that: The reference speech is converted into a Mel spectrogram, and the Mel spectrogram is input into a style encoding module, and the style encoding module extracts style features, including: Performing a preprocessing operation on the reference speech, wherein the processing operation includes a denoising operation and a volume normalization operation; performing a pre-emphasis filtering operation on the reference speech after the pre-processing operation; Inputting the pre-emphasized filtered reference speech into a short-time Fourier transform module to obtain a spectrum representation of the reference speech; The frequency spectrum representation is converted into a Mel spectrum by a Mel filter bank to generate a Mel spectrogram; The Mel spectrum is input into a style encoding module, and the style encoding module analyzes the Mel spectrum using deep learning and extracts style features from the Mel spectrum.

6. The multi-module collaborative speech generation method according to claim 1, characterized in that: Inputting the phoneme sequence and the Mel spectrogram into a text alignment module, and generating alignment information by the text alignment module, includes: Input the phoneme sequence and Mel spectrogram into a text alignment module for alignment processing; Performing frame-level alignment on the phoneme sequence and the Mel spectrogram by using the text alignment module; According to the frame-level alignment result, a time correspondence table between each phoneme and the corresponding audio frame is generated; Based on the time correspondence table, determining the alignment position of each phoneme in the Mel spectrogram; The alignment positions of all phonemes in the Mel spectrogram are integrated to generate complete alignment information.

7. The multi-module collaborative speech generation method according to claim 1, characterized in that: The duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature are input into a decoder to generate a reconstructed Mel spectrogram, including: Inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to control the generation process of the Mel spectrogram; Adjusting the rhythmic features and timbre features in the Mel spectrogram generated by the decoder based on the style features; Adjust the temporal features of the Mel spectrogram generated by the decoder based on the predicted speech duration; Adjust the pitch characteristics of the Mel spectrogram generated by the decoder based on the predicted fundamental frequency; Generate the final adjusted reconstructed Mel spectrum.

8. A multi-module collaborative speech generation device, characterized in that: The multi-module collaborative speech generation device comprises: A data acquisition module, used to acquire reference text and reference speech; A text encoding module, used for converting the reference text into a phoneme sequence, and inputting the phoneme sequence into the text encoding module to generate latent features; A style encoding module, used for converting the reference speech into a Mel spectrogram, and inputting the Mel spectrogram into the style encoding module, and the style encoding module extracts style features; A fundamental frequency extraction module, used for extracting fundamental frequency information from the Mel spectrogram through the fundamental frequency extraction module; A text alignment module, used for inputting the phoneme sequence and the Mel spectrogram into the text alignment module, and the text alignment module generates alignment information; A duration prediction module, configured to predict the duration of the generated speech through the duration prediction module based on the alignment information and the latent features; A fundamental frequency prediction module, configured to predict the fundamental frequency of the generated speech based on the fundamental frequency information through the fundamental frequency prediction module; A decoder module, used for inputting the duration of the predicted generated speech, the fundamental frequency of the predicted generated speech and the style feature into a decoder to generate a reconstructed Mel spectrogram; The vocoder module is used to convert the reconstructed Mel spectrogram into speech audio data using a vocoder.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a multi-module collaborative speech generation program stored in the memory and executable on the processor. When the multi-module collaborative speech generation program is executed by the processor, the steps of the multi-module collaborative speech generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a multi-module collaborative speech generation program, which, when executed by a processor, implements the steps of the multi-module collaborative speech generation method as described in any one of claims 1-7.

Images