Speech synthesis method and device based on rhythm curve, equipment and medium

Abstract

The invention relates to the technical field of speech synthesis, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a speech synthesis method, device and equipment based on a rhythm curve and a medium, and the method comprises the following steps: extracting a text in an original speech sample and a speech signal corresponding to the text; converting the text into a phoneme sequence, and performing duration analysis on the phoneme sequence to obtain voice duration corresponding to each phoneme; extracting a rhythm curve of the voice signal, and performing rhythm analysis on the phoneme sequence, the voice duration and the rhythm curve to obtain a rhythm feature corresponding to the voice signal; obtaining a latent variable, and performing Mel spectrum analysis on the rhythm characteristics and the latent variable to generate a Mel spectrum; and carrying out loss calculation on the generated Mel spectrum and a preset test Mel spectrum, and optimizing the Mel spectrum. The voice generated through model analysis is high in naturalness and rich in rhythm change, the generated voice is more natural and diversified in the aspects of pitch, energy, rhythm and the like, and the requirements of different users can be better met.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular, to a speech synthesis method, device, equipment and medium based on a prosody curve. Background Art

[0002] With the popularization of artificial intelligence and speech interaction technologies, the demand for natural and fluent speech synthesis in various industries has been increasing continuously, which has promoted the development of speech synthesis technology based on prosody curves. Speech synthesis technology is a technology that converts text information into audible speech. Currently, there are many speech synthesis technologies with human-machine speech interaction functions on the market. It interacts with users in natural speech and provides services such as information query, speech broadcast, and task execution.

[0003] In medical training in the field of healthcare, virtual patients can be created through speech synthesis based on prosody curves to simulate speech performances under various conditions for medical students to practice diagnosis.

[0004] In the field of fintech business, there are diverse speech interaction scenarios in fintech. Different business scenarios may require voices with different emotional tendencies. By adjusting the speech prosody, the enthusiasm and positivity of recommendations can be reflected, and at the same time, the loudness can be appropriately increased to highlight key revenue data, accurately conveying the attractiveness of the product.

[0005] In existing speech generation systems, reference audio is usually used as a prosody cue to guide speech synthesis, or natural language descriptions are used to control the generation process. However, the prosody features generated by these two methods usually show consistency in the overall trend, but the changes and emphasis on details are insufficient, resulting in limitations in the expressiveness and diversity of the synthesized speech.

[0006] Therefore, the current technology has significant deficiencies when it comes to generating diverse, delicate and expressive speech. Summary of the Invention

[0007] The present invention provides a speech synthesis method, device, equipment and medium based on a prosody curve. The speech generated through model analysis is not only highly natural, but also has rich prosody variations. The generated speech is more natural and diverse in terms of pitch, energy and rhythm, and can better meet the needs of different users.

[0008] In a first aspect, a speech synthesis method based on a prosody curve is provided, including:

[0009] Extracting the text and the speech signal corresponding to the text in the original speech sample;

[0010] Converting the text into a phoneme sequence and performing duration analysis on the phoneme sequence to obtain the speech duration corresponding to each phoneme;

[0011] Extract the prosody curve of the speech signal, and perform prosody analysis on the phoneme sequence, speech duration, and prosody curve to obtain the prosody features corresponding to the speech signal;

[0012] Obtain latent variables, and perform Mel-spectrum analysis on the prosody features and the latent variables to generate a Mel spectrum;

[0013] Calculate the loss between the generated Mel spectrum and a preset test Mel spectrum to optimize the Mel spectrum;

[0014] Perform speech synthesis on a pre-obtained sample text and the optimized Mel spectrum to obtain synthesized speech.

[0015] In a second aspect, a speech synthesis device based on a prosody curve is provided, including:

[0016] An extraction module for extracting the text in an original speech sample and the speech signal corresponding to the text;

[0017] A conversion module for converting the text into a phoneme sequence;

[0018] A duration analysis module for performing duration analysis on the phoneme sequence to obtain the speech duration corresponding to each phoneme;

[0019] A prosody curve extraction module for extracting the prosody curve of the speech signal;

[0020] A prosody analysis module for performing prosody analysis on the phoneme sequence, speech duration, and prosody curve to obtain the prosody features corresponding to the speech signal;

[0021] An acquisition module for acquiring latent variables;

[0022] A Mel-spectrum analysis module for performing Mel-spectrum analysis on the prosody features and the latent variables to generate a Mel spectrum;

[0023] A loss calculation module for calculating the loss between the generated Mel spectrum and a preset test Mel spectrum to optimize the Mel spectrum;

[0024] A speech synthesis module for performing speech synthesis on a pre-obtained sample text and the optimized Mel spectrum to obtain synthesized speech.

[0025] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned speech synthesis method based on a prosody curve are implemented.

[0026] Fourthly, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned voice synthesis method based on a prosody curve are implemented.

[0027] In the solutions implemented by the above-mentioned voice synthesis method, device, computer device and storage medium based on a prosody curve, compared with traditional TTS models, the voice generated by the model proposed in the present invention not only has high naturalness, but also has rich prosody variations. The generated voice is more natural and diverse in terms of pitch, energy and rhythm, etc., and can better meet the needs of different users. At the same time, thanks to the efficient design of the voice synthesis model and VAE, the model can quickly generate voice from the prosody curve and text input. This feature enables the system to not only meet the real-time voice generation requirements, but also ensure the naturalness and fluency of the generated voice, and is suitable for voice synthesis tasks in dynamic scenarios. At the same time, through the degree of difference between the Mel spectrogram generated by the model and the preset test Mel spectrogram, researchers can intuitively judge the performance of the model in the task of generating Mel spectrograms, and then objectively compare different models or different parameter settings of the same model, providing an important basis for the selection and optimization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1 is a schematic diagram of an application environment of a voice synthesis method based on a prosody curve in an embodiment of the present invention;

[0030] Figure 2 is a schematic flowchart of a voice synthesis method based on a prosody curve in an embodiment of the present invention;

[0031] Figure 3 is a schematic structural diagram of a voice synthesis device based on a prosody curve in an embodiment of the present invention;

[0032] Figure 4 is a processing flowchart of a voice synthesis method based on a prosody curve in an embodiment of the present invention;

[0033] Figure 5 is a schematic structural diagram of a computer device in an embodiment of the present invention;

[0034] Figure 6 is another schematic structural diagram of a computer device in an embodiment of the present invention. Specific Embodiments

[0035] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0036] A method for speech synthesis based on a prosody curve provided by an embodiment of the present invention can be applied to an application environment such as Figure 1 In this application environment, the client communicates with the server through a network. The server can extract the text in the original speech sample and the speech signal corresponding to the text; convert the text into a phoneme sequence, perform duration analysis on the phoneme sequence to obtain the speech duration corresponding to each phoneme; extract the prosody curve of the speech signal, and perform prosody analysis on the phoneme sequence, speech duration, and prosody curve to obtain the prosody features corresponding to the speech signal; obtain latent variables, and perform Mel spectrum analysis on the prosody features and the latent variables to generate a Mel spectrum; calculate the loss between the generated Mel spectrum and a preset test Mel spectrum to optimize the Mel spectrum; perform speech synthesis on a pre-acquired sample text and the optimized Mel spectrum to obtain a synthesized speech, and feedback the synthesized speech to the client. The present invention provides a speech synthesis device based on a prosody curve. For the synthesized speech service, the speech generated through model analysis is not only highly natural, but also has rich prosody variations. The generated speech is more natural and diverse in terms of pitch, energy, rhythm, etc., and can better meet the needs of different users. Among them, the client can be, but is not limited to, various personal computers, laptop computers, smartphones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail through specific embodiments below.

[0037] Please refer to Figure 2 as shown in Figure 2 which is a flowchart of a method for speech synthesis based on a prosody curve provided by an embodiment of the present invention, including the following steps:

[0038] S1. Extract the text in the original speech sample and the speech signal corresponding to the text.

[0039] In the embodiment of the present invention, the extraction refers to intercepting the content of the text from the original speech sample, and according to the content of the text, locating and intercepting the speech signal segment matching the text from the original speech sample.

[0040] Specifically, the original speech sample is converted into text using a speech recognition model, and then a text-audio alignment tool is used to align the text with the speech signal in terms of time. According to the alignment result, the speech signal segment (such as a time interval) corresponding to the text is found, and the extracted speech signal segment is saved as a separate speech signal file.

[0041] In a specific medical and health scenario, when a doctor communicates with a patient about the patient's condition, the system extracts the text in the speech sample in real time and automatically generates a medical record, improving the efficiency of medical record entry and reducing manual entry errors. For example, when the patient describes "always feeling headache in the past week and sometimes dizzy", the system can accurately extract and record it. At the same time, the speech text in expert lectures and case discussions can be extracted and organized into teaching materials for the convenience of medical staff to learn.

[0042] In a specific fintech scenario, by extracting the speech text of customers when consulting financial products, the needs and preferences of customers can be mined. For example, when a customer asks about the returns of equity funds, it can be judged that the customer has a certain interest in stock investment, and then relevant products can be recommended. At the same time, by combining speech signals and text data, big data analysis can be carried out to understand the behavior patterns of customer groups and market trends, providing a basis for the design of financial products and marketing strategies.

[0043] In the embodiments of the present invention, the extraction of the text in the original speech sample and the speech signal corresponding to the text includes:

[0044] Converting the speech content in the original speech sample into initial text;

[0045] Dividing the initial text into multiple text segments according to preset conditions;

[0046] Performing audio alignment on the text segments according to the original speech sample to obtain the speech signal corresponding to each text segment.

[0047] In the embodiments of the present invention, the conversion refers to converting the original speech sample into text through speech recognition technology, the division refers to splitting a continuous text into multiple independent text segments according to specific rules or criteria, and the audio alignment refers to matching the text with the speech signal in terms of time to determine the specific time positions of each word or each phoneme in the text in the speech signal.

[0048] Specifically, use a speech recognition model to convert audio into text, add punctuation marks to the text, and adjust the text format, such as text case, paragraphing, etc.; set the conditions for dividing the text, such as punctuation marks, sentence length, etc., divide the text according to the set conditions to obtain multiple groups of text, and then match the text with the audio in terms of time to determine the specific time positions of each word or phoneme in the text in the audio. An alignment tool or model (such as dynamic time warping, hidden Markov model, deep learning model) can be used for alignment.

[0049] In the embodiments of the present invention, the text in the original speech sample and the speech signal corresponding to the text are extracted to facilitate the speech assistant to understand and execute user instructions, convert the speech instruction into text to facilitate device control, improve the convenience and accuracy of human-computer interaction, improve the training effect and generalization ability of the model, improve the quality and efficiency of speech synthesis, generate higher-quality speech, and at the same time improve the user experience, can generate diverse speech outputs to meet different scenario requirements, improve the accuracy of the speech recognition model, and at the same time support speech annotation and analysis to provide high-quality data support for speech analysis tasks.

[0050] In the embodiments of the present invention, extracting the text in the original speech sample can improve the information processing efficiency, facilitate subsequent text analysis and data mining, and improve the convenience and accuracy of human-computer interaction. At the same time, add text annotations to the speech signal for training the speech recognition model or speech synthesis model. Extracting the speech signal corresponding to the text can be used in the speech synthesis task to train the speech synthesis model and provide high-quality training data to improve the speech synthesis effect.

[0051] S2. Convert the text into a phoneme sequence, and perform duration analysis on the phoneme sequence to obtain the speech duration corresponding to each phoneme.

[0052] In the embodiments of the present invention, the conversion refers to decomposing each word or character in a text into corresponding phonemes to obtain the phoneme representation of the text. The duration analysis refers to determining the duration of each phoneme in the phoneme sequence in the speech signal by analyzing the original speech sample.

[0053] Specifically, convert numbers, abbreviations, etc. in the text into standard forms, split the text into words or phrases, use a dictionary or rules to convert the words into a phoneme sequence. For polyphonic words, the correct pronunciation needs to be determined according to the context. Use a statistical model (such as HMM or DNN) to predict the duration of each phoneme, consider factors such as context and speech rate, adjust the phoneme duration according to the speech synthesis requirements to match the natural speech rhythm, and finally generate a phoneme sequence with duration information.

[0054] In an embodiment of the present invention, the duration analysis of the phoneme sequence to obtain the speech duration corresponding to each phoneme includes:

[0055] Perform noise reduction processing on the phoneme sequence one by one to obtain the noise-reduced speech of each phoneme;

[0056] Divide the noise-reduced speech into multiple groups of frame segment speeches;

[0057] Perform duration feature calculation on the frame segment speeches one by one to obtain the duration features corresponding to the frame segment speeches;

[0058] Generate the speech duration corresponding to each phoneme according to the combination of the duration features corresponding to the frame segment speeches.

[0059] In an embodiment of the present invention, the noise reduction processing refers to removing noise from the speech signal of the phoneme sequence to improve the clarity and quality of the speech. The division refers to dividing a continuous noise-reduced speech into multiple groups of frame segment speeches according to preset conditions. The duration feature calculation refers to analyzing the frame segment speeches and extracting their distribution features in time. The combination generation refers to splicing and adding the duration features corresponding to multiple frame segment speeches.

[0060] Specifically, since the collected speech signal may contain interference such as environmental noise, a noise reduction algorithm needs to be used to remove the noise. For example, a noise reduction method based on wavelet transform can decompose the speech signal into different wavelet domains and remove the noise components through threshold processing; or the spectral subtraction method can be used to estimate the noise spectrum and subtract it from the noisy speech spectrum. The processed speech signal is divided into several short frames, with the frame length generally being 20 - 30 milliseconds and the frame shift usually being 10 milliseconds to ensure a certain overlap between adjacent frames for subsequent analysis. Calculate the acoustic features for each frame, such as Mel Frequency Cepstral Coefficients (MFCC), Linear Predictive Cepstral Coefficients (LPCC), etc. Taking MFCC as an example, first perform a Fast Fourier Transform (FFT) on each frame to obtain the spectrum, then convert the spectrum to the Mel frequency domain through a Mel filter bank, take the logarithm and perform a Discrete Cosine Transform (DCT) to obtain the MFCC coefficients. Use the Viterbi algorithm, etc. to find the phoneme sequence path that best matches the input feature sequence in the acoustic model, so as to determine the boundaries of each phoneme and the corresponding acoustic features. In the Viterbi algorithm, the optimal path of each state at each time point is calculated through dynamic programming, and finally the optimal segmentation of the entire phoneme sequence is obtained. After determining the phoneme boundaries, perform statistical frame segment speech duration feature calculation, and finally calculate the duration of the phoneme.

[0061] Specifically, for example, after text preprocessing, input "cat", and through phoneme conversion to The predicted duration is Finally, generate a phoneme sequence with duration

[0062] Furthermore, in the synthetic speech analysis in the field of medical and health, speech synthesis technology can be used to reply to patients' consultation questions. This type of autonomous speech interaction method obtains synthetic speech according to text and prosody. However, at present, all the content replied by this speech interaction method when replying to patients' questions is full of mechanical feeling, resulting in a poor interaction experience. However, through the analysis of phoneme duration, it can provide a basis for subsequent speech synthesis and improve the user experience.

[0063] Similarly, when applying speech synthesis technology to the business in the financial field and using speech technology to broadcast financial-related content, the speech recognition system sometimes has difficulty correctly identifying the key information in the content, making the whole report mechanical in broadcast without any emotional ups and downs, which is likely to cause customers to miss important information and bring economic losses to customers; through the analysis of phoneme duration, the accuracy and reliability of speech recognition can be improved, and at the same time, the professionalism of speech synthesis can also be improved through noise reduction, volume adjustment, sentence segmentation, etc.

[0064] In the embodiment of the present invention, noise reduction processing is performed on the phoneme sequence, which can effectively remove these noises, make the speech of each phoneme clearer and purer, improve the overall speech quality, and contribute to the subsequent analysis and processing of speech. The denoised speech is divided into frame-segmented speech, which can discretize the continuous speech signal, facilitate detailed feature extraction and analysis of each frame segment. At the same time, compared with performing a one-time processing on the entire speech signal, dividing it into multiple frame segments for processing can reduce the computational complexity. By calculating the duration features of each frame-segmented speech, more detailed and accurate speech duration information can be obtained. Not only can the approximate duration of the entire phoneme be obtained, but also the duration distribution of the phoneme in different frame segments can be understood, which helps to more deeply analyze the prosody structure and pronunciation characteristics of the speech. Combining the duration features of each frame-segmented speech can comprehensively consider the duration information of the phoneme in different local areas, avoiding the information loss and inaccuracy problems caused by only relying on a single frame segment or simple average duration. It can comprehensively describe the duration characteristics of the phoneme from multiple dimensions, making the generated phoneme speech duration more representative and accurate.

[0065] In the embodiment of the present invention, converting text into a phoneme sequence and analyzing the duration can provide richer and more accurate feature information for the speech recognition system, help the system more accurately distinguish similar phonemes, thereby improving the accuracy of speech recognition. Duration analysis can enable the synthetic speech to reasonably adjust the phoneme duration according to factors such as the semantics and emotions of the text, so as to accurately express prosodic features such as stress and intonation, making the synthetic speech more vivid and better able to convey the emotions and intentions of the text.

[0066] S3. Extracting the prosody curve of the speech signal, and performing prosody analysis on the phoneme sequence, speech duration and prosody curve to obtain prosody features corresponding to the speech signal.

[0067] In an embodiment of the present invention, the extraction refers to analyzing and obtaining prosody-related features from a speech signal, such as curves of pitch, energy, duration, etc. that change over time, and the prosody analysis refers to systematically studying and analyzing the prosody features in a speech signal to extract information related to speech rhythm, intonation, stress, etc.

[0068] Specifically, fundamental frequency, duration, energy, etc. are generally used as basic parameters of prosodic features. Commonly used methods such as autocorrelation method and cepstrum method are used to extract fundamental frequency information and obtain fundamental frequency curve; energy information is obtained by analyzing the amplitude of the signal to form an energy curve; these curves constitute the main part of the prosodic curve, and the phoneme sequence is aligned with the prosodic curve in time, and the prosodic curve segment corresponding to each phoneme is observed to analyze the changes in its fundamental frequency, energy and other prosodic parameters. For example, it is determined whether the phoneme is in the stage of rising, falling or stable fundamental frequency, whether the energy is strong or weak, etc., so as to determine the prosodic characteristics of the phoneme, such as whether it is stressed, whether the tone is rising or falling, etc. The analysis results of the phoneme sequence, speech duration and prosodic curve are statistically analyzed and modeled, and the average value, standard deviation, maximum value, minimum value and other statistical quantities of various prosodic parameters can be calculated to summarize the overall prosodic features. At the same time, some methods in machine learning or deep learning, such as hidden Markov model (HMM) and deep neural network (DNN), can also be used to model these rhythmic features and explore more complex rhythmic patterns and rules.

[0069] In the embodiment of the present invention, the step of extracting the prosody curve of the speech signal includes:

[0070] Converting the speech signal into a time domain signal, and performing Fourier transform on the time domain signal to obtain a frequency domain signal;

[0071] Performing pitch curve calculation on the frequency domain signal to obtain a pitch curve;

[0072] Calculate the short-time energy of the speech signal and obtain the sound intensity curve;

[0073] A prosody curve is generated according to the combination of the pitch curve and the intensity curve.

[0074] In the embodiments of the present invention, the conversion refers to converting a voice signal from its original form into a time-domain representation, i.e., the waveform of the voice signal over time. The Fourier transform refers to a mathematical tool for converting a time-domain signal into a frequency-domain signal. The pitch curve calculation refers to extracting the curve of how pitch changes over time from the frequency-domain signal. The calculation refers to dividing the voice signal into multiple short time periods (frames) and calculating the energy value of each period to obtain the intensity curve. The combined generation refers to integrating the pitch curve and the intensity curve to generate a comprehensive prosody curve.

[0075] Specifically, voice signals usually exist in analog form in nature. Through audio acquisition devices such as microphones, the pressure wave of sound is converted into an electrical signal, which is the time-domain signal. The representation of the voice signal in frequency usually appears in the form of a curve of amplitude varying with frequency, which is usually called the frequency-domain signal. The process of converting the voice signal into a time-domain signal and performing the Fourier transform on the time-domain signal to obtain the frequency-domain signal includes steps such as loading the voice signal, converting it into a time-domain signal, performing the Fourier transform, and outputting the frequency-domain signal. Common tools include librosa, pydub, NumPy, and scipy, etc.

[0076] Furthermore, extracting the fundamental frequency from the frequency-domain signal is the key to calculating the pitch curve. Common methods include the autocorrelation method, the cepstrum method, etc. The fundamental frequency is extracted for each frame of the frequency-domain signal to obtain a series of fundamental frequency values. Arranging these fundamental frequency values in chronological order can obtain the curve of how pitch changes over time, i.e., the pitch curve. Usually, some smoothing processing is performed on the fundamental frequency values to remove possible noise and outliers, making the pitch curve smoother and more stable. To calculate the short-time energy of the voice signal, first, the voice signal needs to be divided into several short frames. Usually, window functions such as the Hamming window and the rectangular window are used to window the signal to reduce problems such as spectral leakage. After calculating the short-time energy of each frame, arranging these energy values in chronological order can obtain the curve of how the intensity of the voice signal changes over time, i.e., the intensity curve. Similarly, to make the curve smoother, some smoothing filtering processing may be performed on the energy values.

[0077] Specifically, subsequently, the pitch curve and the intensity curve are reasonably combined. First, they need to be normalized to map their values to the same value range, such as [0, 1]. According to the selected combination method, the prosody curve is calculated. If necessary, further adjustment and optimization can be performed on the prosody curve, such as smoothing processing and removing outliers, etc., to make the prosody curve more accurately reflect the prosody characteristics of the voice signal.

[0078] In the synthetic speech analysis in the field of medical and health, by analyzing the prosody curve, speech disorders (such as dysarthria, aphasia, etc.) can be identified to assist doctors in diagnosis. For example, in disease monitoring, certain diseases (such as Parkinson's disease, Alzheimer's disease) can affect the speech characteristics of patients, and the progression of the disease can be monitored through prosody curve analysis. For patients with language disorders, through the analysis of prosody curves, personalized speech rehabilitation training programs can be designed. At the same time, by comparing the prosody curves before and after treatment, the effect of rehabilitation treatment can be evaluated.

[0079] In the financial voice interaction scenario, accurately identifying the user's speech is crucial. The prosody curve provides additional feature information, which helps to distinguish between words or phrases with similar pronunciations. For example, for "interest rate increase" and "benefit increase", by analyzing the prosody features, it is possible to more accurately identify whether the user is saying "interest rate" or "benefit" during voice interaction, reducing recognition errors caused by similar pronunciations and improving the accuracy of the speech recognition system.

[0080] In the embodiments of the present invention, the speech signal usually appears as a waveform that changes with time in the time domain, and it is difficult to directly see its frequency components. After being transformed to the frequency domain through Fourier transform, it can clearly show which sine waves of different frequencies the signal is composed of and the relative intensities of each frequency component, helping to analyze the spectral characteristics of the sound. Calculating the pitch curve can intuitively present the change of the fundamental frequency of the speech signal over time, and calculating the intensity curve can show the dynamic change range and trend of the sound, helping to analyze the stability, mutation situation, etc. of the sound. The prosody curve generated by combining the two synthesizes the change information of pitch and intensity, and can more comprehensively and accurately describe the prosody characteristics of the sound.

[0081] In the embodiments of the present invention, as an important attribute of speech, the prosody feature can provide an additional information dimension for the speech recognition system. Different speech contents are often accompanied by specific prosody patterns. Extracting the prosody curve helps to capture these patterns and distinguish between words or sentences with similar pronunciations, thereby improving the accuracy of speech recognition. Combining the prosody curve with other speech features (such as spectral features, etc.) can perform a more comprehensive and in-depth analysis of the emotion in the speech from multiple dimensions.

[0082] S4. Obtain a latent variable, and perform Mel spectrum analysis on the prosody feature and the latent variable to generate a Mel spectrum.

[0083] In the embodiments of the present invention, the process of obtaining a latent variable extracted or generated from data, and the Mel spectrum analysis refers to the process of converting a speech signal into a spectral representation based on the Mel scale.

[0084] Specifically, the input data (such as Mel spectrogram) is mapped to mean and variance by an encoder, the latent variable is sampled through the reparameterization trick, the prosody feature vector is concatenated with the latent variable to form a joint feature vector, and then a conditional generation model (such as Tacotron, FastSpeech, VITS) is used to generate the Mel spectrogram. The input prosody feature plus the latent variable outputs a sequence of Mel spectrogram frames.

[0085] In the embodiments of the present invention, the obtaining of the latent variable includes:

[0086] Performing feature convolution on the pre-obtained Mel spectrogram sample data to obtain convolution Mel spectrogram data;

[0087] Performing feature pooling on the convolution Mel spectrogram data to obtain pooled Mel spectrogram data;

[0088] Mapping the pooled Mel spectrogram data to a mean vector in the preset latent space to obtain the mean vector of the latent space distribution;

[0089] Mapping the pooled Mel spectrogram data to a logarithmic variance vector in the preset latent space to obtain the logarithmic variance vector of the latent space distribution;

[0090] Performing linear programming on the logarithmic variance vector and a preset random vector to obtain a programmed vector;

[0091] Combining the mean vector and the programmed vector to generate a latent variable.

[0092] In the embodiments of the present invention, the feature convolution refers to applying a convolution operation to the Mel spectrogram sample data to extract higher-level feature representations and obtain convolution Mel spectrogram data. The feature pooling refers to performing a downsampling operation on the convolution Mel spectrogram data to reduce the data dimension and retain important features, thereby obtaining pooled Mel spectrogram data. The mean vector mapping refers to mapping high-dimensional data to a low-dimensional latent space and extracting its statistical features (such as mean, logarithmic variance). The linear programming refers to calculating the logarithmic variance vector and a preset random vector using element-wise multiplication.

[0093] Specifically, an encoder part of a variational autoencoder (VAE) is constructed, which is usually composed of multiple convolutional layers and pooling layers. The convolutional layer is used to extract local features of the input Mel spectrogram sample data, and the pooling layer is used to reduce the data dimension while retaining important feature information. For example, a 3×3 convolutional kernel is used for convolution operation, the stride is set to 1, and the padding is 1 to keep the size of the feature map unchanged. Then, a 2×2 pooling kernel is used for max pooling operation with a stride of 2 to reduce the data dimension. After the convolutional layer and the pooling layer, a fully connected layer is added. The fully connected layer maps the extracted features to the latent space and outputs two vectors, which are the mean vector μ of the latent space distribution and the logarithmic variance vector logσ respectively.2 , assuming that the dimension of the feature vector obtained after the convolutional and pooling operations on the input Mel spectrogram is d1, and the dimension of the latent space is d2, then the dimension of the weight matrix of the fully connected layer is d1 * d2. Subsequently, the mean vector μ and the log variance vector logσ output by the encoder 2 .

[0094] Furthermore, using the reparameterization trick, a random vector ε is sampled from the standard normal distribution N(0, I), where I is the identity matrix. The dimension of ε is the same as that of the latent space, assumed to be d2. According to the mean vector μ and the log variance vector logσ output by the encoder 2 , the latent variable can be calculated by the following formula:

[0095]

[0096] where z refers to the latent variable, σ refers to exp=(0.5 * logσ 2 ), refers to element-wise multiplication.

[0097] In the embodiments of the present invention, the performing Mel spectrogram analysis on the prosodic features and the latent variable to generate a Mel spectrogram includes:

[0098] Performing feature fusion on the prosodic features and the latent variable to obtain a fusion result;

[0099] Mapping the fusion result to a pre-obtained Mel spectrogram space to obtain a Mel spectrogram.

[0100] In the embodiments of the present invention, the feature fusion refers to integrating the prosodic features and the latent variable to generate a comprehensive fusion result to fully utilize the information of both and improve the performance of the model. The mapping refers to the process of aligning or matching the fusion result with the features in the Mel spectrogram space to generate a new Mel spectrogram.

[0101] Specifically, first align the time axes of the prosodic features and the latent variable, then perform normalization processing on the prosodic features and the latent variable, and finally integrate the prosodic features and the latent variable, and connect the integrated features in chronological order to generate a fusion result; align the features with the same time resolution and frame length, align the fusion result with the features in the Mel spectrogram space to ensure their consistency in time, and then convert the fusion result to the Mel spectrogram space to generate a new Mel spectrogram.

[0102] In the synthetic speech analysis in the field of healthcare, concatenating the prosodic feature vectors with the vectors obtained by sampling in the latent space to generate the Mel spectrogram can make the synthetic speech more natural and clear, and improve the patient's understanding of voice instructions, information announcements, etc. For example, in an intelligent medical guidance system, clear and natural synthetic speech can better guide patients to complete various examination processes, reducing the anxiety and confusion of patients caused by not understanding the voice prompts.

[0103] There are diverse voice interaction scenarios in fintech. Different business scenarios may require voices with different emotional tendencies. By concatenating the prosodic feature vectors with the vectors obtained by sampling in the latent space to generate the Mel spectrogram, the emotion of the synthetic speech can be shaped and adjusted from multiple dimensions. For example, using a warm and positive voice emotion when recommending products to customers and a patient and soothing emotion when handling customer complaints, thereby improving the user's satisfaction with voice interaction.

[0104] In the embodiments of the present invention, the convolution operation is calculated by sliding a convolution kernel on the Mel spectrogram, which can effectively capture these local features. The pooling operation can, to a certain extent, highlight more important features, suppress noise and unimportant details, helping to retain the most significant feature information, enabling the model to focus more on the features that are more meaningful for classification or other tasks, and improving the robustness and discriminability of the features. The mean vector represents the average position of the data, while the log variance vector reflects the degree of dispersion of the data in each dimension. The combination of the two can more comprehensively describe the statistical characteristics of the data, enabling the model to not only output a definite result but also give an estimate of the uncertainty of this result.

[0105] In the embodiments of the present invention, the generated different Mel spectrograms can provide richer and more diverse prosody and timbre choices for the synthetic speech, making the synthetic speech more natural, vivid, closer to real human speech, improving the quality of speech synthesis and the user experience, and having important application value in fields such as intelligent customer service and voice assistants. Through the Mel spectrogram analysis, different Mel spectrograms can more effectively extract these emotional features, providing stronger support for speech emotion analysis, and helping to develop intelligent voice systems that can understand and respond to human emotions, which are applied in fields such as mental health monitoring and intelligent companionship.

[0106] S5. Calculate the loss between the generated Mel spectrogram and a preset test Mel spectrogram to optimize the Mel spectrogram.

[0107] In the embodiments of the present invention, the loss calculation refers to comparing the Mel spectrogram with a preset test Mel spectrogram and calculating the difference or error between the two.

[0108] Specifically, by using the prosodic features and the mel spectrogram generated by the latent variables, and setting a test mel spectrogram, which is usually used as a reference or a target, calculate the difference or error between the mel spectrogram and the test mel spectrogram, and evaluate the performance of the model or optimize the parameters of the model through loss calculation.

[0109] In the embodiments of the present invention, the calculating the loss between the generated mel spectrogram and a preset test mel spectrogram to optimize the mel spectrogram includes:

[0110] Calculating a loss value between the mel spectrogram and the preset test mel spectrogram;

[0111] Adjusting the mel spectrogram according to the loss value to obtain an optimized mel spectrogram.

[0112] In the embodiments of the present invention, the calculating refers to the process of calculating the difference or error between the mel spectrogram and the preset test mel spectrogram, and the adjusting refers to the process of integrating the loss result with the mel spectrogram to generate the calculated mel spectrogram.

[0113] Specifically, common loss functions for measuring the difference between two mel spectrograms include mean squared error (MSE), mean absolute error (MAE), etc. If more attention is paid to the overall degree of difference, MSE is a suitable choice, which calculates the average of the squares of the differences of the corresponding elements of the two mel spectrograms; if one wants to highlight the points with larger individual differences more, MAE may be more suitable, which calculates the average of the absolute values of the differences of the corresponding elements; ensure that the two mel spectrograms for calculating the loss are aligned in dimension and time scale. The mel spectrogram is usually a two-dimensional matrix, and the dimension may represent mel frequency bins and time frames. If the dimensions or time lengths of the two mel spectrograms are inconsistent, adjustment is required, which may involve cropping or padding operations to make them have the same shape for accurate element-by-element comparison.

[0114] Further, taking the mean squared error loss function as an example, assume that the mel spectrogram M and the preset test mel spectrogram M ref are both matrices of m*n, where m is the number of mel frequencies and n is the number of time frames. The calculation method of the loss L is:

[0115]

[0116] where, represents the test mel spectrogram data in the i-th row and j-th column of the matrix of the test mel spectrogram M ref , and M ij represents the mel spectrogram data in the i-th row and j-th column of the matrix of the mel spectrogram M.

[0117] Specifically, the processed loss matrix is directly added to the original Mel spectrum to obtain the calculated Mel spectrum, where is the processed loss matrix. In this way, each element of the Mel spectrum is affected by the loss. The greater the loss, the more obvious the change to the Mel spectrum.

[0118] In the embodiments of the present invention, the loss result reflects the difference between the current Mel spectrum and the preset Mel spectrum. By combining it with the Mel spectrum, information related to the preset Mel spectrum can be introduced into the Mel spectrum, making the Mel spectrum closer to the ideal state, highlighting some important features, or correcting some possible deviations. At the same time, the fusion of loss information and Mel spectrum data is achieved, enriching the information contained in the Mel spectrum.

[0119] In the embodiments of the present invention, the result of loss calculation can clearly and quantitatively reflect the degree of difference between the Mel spectrum generated by the model and the preset test Mel spectrum. Through this value, researchers can intuitively judge the performance of the model in the task of generating Mel spectra, and then objectively compare different models or different parameter settings of the same model, providing an important basis for model selection and optimization.

[0120] S6. Perform voice synthesis on the pre-acquired sample text and the calculated Mel spectrum to obtain synthesized speech.

[0121] Specifically, the calculated Mel spectrum contains the energy distribution information of the audio on the Mel frequency scale. Analyze the characteristics of the Mel spectrum and extract key information related to voice synthesis, such as the energy changes in different frequency bands, the energy dynamics in the time series, etc. These information will directly affect the characteristics of the synthesized speech, such as timbre, pitch, and rhythm. Convert the Mel spectrum information into more specific acoustic parameters, such as fundamental frequency (F0), formant frequency, spectral envelope, etc. Different Mel spectrum energy distributions are different, and the acoustic parameters are different. When the same text is input, the combination of the text with different Mel spectra can generate synthesized speech with different rhythms.

[0122] In the analysis of medical and health voice synthesis, for medical education, the voice synthesis technology of combining text with different Mel spectra can convert text content such as medical textbooks and courseware into vivid voice explanations, cooperate with materials such as images and videos, provide more abundant learning resources for medical students, help them learn and understand medical knowledge more efficiently, and improve learning effects.

[0123] In real life, the customers of fintech services may come from all over the world. Through this voice synthesis technology, the sample text can be conveniently converted into synthesized speech in multiple languages, and it can also adapt to people of multiple age groups, providing personalized services for customers with different language backgrounds and different age groups, expanding the service scope, and enhancing the competitiveness of financial institutions in the market.

[0124] It can be seen that in the above solution, for the synthetic speech service, the text in the original speech sample and the speech signal corresponding to the text are extracted; the text is converted into a phoneme sequence, and the duration analysis is performed on the phoneme sequence to obtain the speech duration corresponding to the phoneme; the prosody curve of the speech signal is extracted, and the phoneme sequence, speech duration, and prosody curve are subjected to prosody analysis to obtain the prosody features corresponding to the speech signal; a latent variable is obtained, and the Mel spectrum analysis is performed on the prosody features and the latent variable to generate a Mel spectrum; the loss calculation is performed on the Mel spectrum and a preset test Mel spectrum to obtain the calculated Mel spectrum; the pre-obtained sample text and the calculated Mel spectrum are used for speech synthesis to obtain synthetic speech. The speech generated through model analysis not only has high naturalness but also has rich prosody variations. The generated speech is more natural and diverse in terms of pitch, energy, and rhythm, and can better meet the needs of different users.

[0125] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0126] In one embodiment, a speech synthesis device based on a prosody curve is provided. This speech synthesis device based on a prosody curve corresponds one-to-one with the speech synthesis method based on a prosody curve in the above embodiment. As Figure 3 shown, this speech synthesis device based on a prosody curve includes an extraction module 101, a conversion module 102, a duration analysis module 103, a prosody curve extraction module 104, a prosody analysis module 105, an acquisition module 106, a Mel spectrum analysis module 107, a loss calculation module 108, and a speech synthesis module 109. The detailed descriptions of each functional module are as follows:

[0127] The extraction module 101 is used to extract the text in the original speech sample and the speech signal corresponding to the text;

[0128] The conversion module 102 is used to convert the text into a phoneme sequence;

[0129] The duration analysis module 103 is used to perform duration analysis on the phoneme sequence to obtain the speech duration corresponding to each phoneme;

[0130] The prosody curve extraction module 104 is used to extract the prosody curve of the speech signal;

[0131] The prosody analysis module 105 is used to perform prosody analysis on the phoneme sequence, speech duration, and prosody curve to obtain the prosody features corresponding to the speech signal;

[0132] The acquisition module 106 is used to acquire a latent variable;

[0133] The Mel-spectrum analysis module 107 is used to perform Mel-spectrum analysis on the prosody features and the latent variables to generate a Mel-spectrum.

[0134] The loss calculation module 108 is used to calculate the loss between the generated Mel-spectrum and a preset test Mel-spectrum to optimize the Mel-spectrum.

[0135] The speech synthesis module 109 is used to perform speech synthesis on a pre-acquired sample text and the optimized Mel-spectrum to obtain synthesized speech.

[0136] In one embodiment, when the extraction module 101 extracts the text and the speech signal corresponding to the text from the original speech sample, it is used to:

[0137] Convert the speech content in the original speech sample into initial text;

[0138] Divide the initial text into multiple text segments according to preset conditions;

[0139] Perform audio alignment on the text segments according to the original speech sample to obtain the speech signal corresponding to each text segment.

[0140] In one embodiment, when the duration analysis module 103 performs duration analysis on the phoneme sequence to obtain the speech duration corresponding to each phoneme, it is used to:

[0141] Perform noise reduction processing on the phoneme sequence one by one to obtain the noise-reduced speech of each phoneme;

[0142] Divide the noise-reduced speech into multiple groups of frame-segmented speech;

[0143] Perform duration feature calculation on the frame-segmented speech one by one to obtain the duration feature corresponding to the frame-segmented speech;

[0144] Generate the speech duration corresponding to each phoneme according to the duration features corresponding to the frame-segmented speech.

[0145] In one embodiment, when the prosody curve extraction module 104 extracts the prosody curve of the speech signal, it is used to:

[0146] Convert the speech signal into a time-domain signal and perform Fourier transform on the time-domain signal to obtain a frequency-domain signal;

[0147] Perform pitch curve calculation on the frequency-domain signal to obtain a pitch curve;

[0148] Calculate the short-time energy of the speech signal to obtain an intensity curve;

[0149] Generate a prosody curve according to the combination of the pitch curve and the intensity curve.

[0150] In one embodiment, when the acquisition module 106 acquires the latent variable, it is used for:

[0151] Perform feature convolution on the pre-acquired Mel-spectrum sample data to obtain convolution Mel-spectrum data;

[0152] Perform feature pooling on the convolution Mel-spectrum data to obtain pooled Mel-spectrum data;

[0153] Map the pooled Mel-spectrum data according to a preset latent space to obtain the mean vector of the latent space distribution;

[0154] Map the pooled Mel-spectrum data according to a preset latent space to obtain the logarithmic variance vector of the latent space distribution;

[0155] Perform linear programming on the logarithmic variance vector and a preset random vector to obtain a planned vector;

[0156] Combine the mean vector and the planned vector to generate a latent variable.

[0157] In one embodiment, when the Mel-spectrum analysis module 107 performs Mel-spectrum analysis on the prosody feature and the latent variable to generate a Mel-spectrum, it is used for:

[0158] Perform feature fusion on the prosody feature and the latent variable to obtain a fusion result;

[0159] Map the fusion result to a pre-acquired Mel-spectrum space to obtain a Mel-spectrum.

[0160] In one embodiment, when the loss calculation module 108 calculates the loss between the generated Mel-spectrum and a preset test Mel-spectrum to optimize the Mel-spectrum, it is used for:

[0161] Calculate the loss value between the Mel-spectrum and the preset test Mel-spectrum;

[0162] Adjust the Mel-spectrum according to the loss value to obtain an optimized Mel-spectrum.

[0163] The present invention provides a voice synthesis device based on a prosody curve. For the synthesized voice service, the text in the original voice sample and the voice signal corresponding to the text are extracted; the text is converted into a phoneme sequence, and the duration analysis is performed on the phoneme sequence to obtain the voice duration corresponding to the phonemes; the prosody curve of the voice signal is extracted, and the phoneme sequence, the voice duration, and the prosody curve are subjected to prosody analysis to obtain the prosody features corresponding to the voice signal; a latent variable is obtained, and Mel spectrograms are generated by performing Mel spectrogram analysis on the prosody features and the latent variable; the loss is calculated between the Mel spectrograms and a preset test Mel spectrogram to obtain the calculated Mel spectrogram; the pre-acquired sample text and the calculated Mel spectrogram are used for voice synthesis to obtain the synthesized voice. The voice generated through model analysis not only has high naturalness but also has rich prosody variations. The generated voice is more natural and diverse in terms of pitch, energy, rhythm, etc., and can better meet the needs of different users.

[0164] For the specific limitations on a voice synthesis device based on a prosody curve, reference can be made to the limitations on a voice synthesis method based on a prosody curve in the above text, which will not be elaborated here. Each module in the above voice synthesis device based on a prosody curve can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in the form of hardware or be independent of it, or can be stored in the memory in the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above modules.

[0165] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 5 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a voice synthesis method based on a prosody curve.

[0166] In one embodiment, a computer device is provided. The computer device can be a client, and its internal structure diagram can be as Figure 6As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of a voice synthesis method based on a prosody curve.

[0167] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are realized:

[0168] Extract the text and the voice signal corresponding to the text in the original voice sample;

[0169] Convert the text into a phoneme sequence, and perform duration analysis on the phoneme sequence to obtain the voice duration corresponding to each phoneme;

[0170] Extract the prosody curve of the voice signal, and perform prosody analysis on the phoneme sequence, voice duration, and prosody curve to obtain the prosody features corresponding to the voice signal;

[0171] Obtain latent variables, and perform Mel spectrum analysis on the prosody features and the latent variables to generate a Mel spectrum;

[0172] Calculate the loss between the generated Mel spectrum and a preset test Mel spectrum to optimize the Mel spectrum;

[0173] Perform voice synthesis on the pre-obtained sample text and the optimized Mel spectrum to obtain a synthesized voice.

[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the following steps are realized:

[0175] Extract the text and the voice signal corresponding to the text in the original voice sample;

[0176] Convert the text into a phoneme sequence, and perform duration analysis on the phoneme sequence to obtain the voice duration corresponding to each phoneme;

[0177] Extract the prosody curve of the voice signal, and perform prosody analysis on the phoneme sequence, voice duration, and prosody curve to obtain the prosody features corresponding to the voice signal;

[0178] Obtain latent variables, perform Mel-spectrum analysis on the prosodic features and the latent variables to generate a Mel spectrum;

[0179] Calculate the loss between the generated Mel spectrum and a preset test Mel spectrum to optimize the Mel spectrum;

[0180] Perform speech synthesis on the pre-obtained sample text and the optimized Mel spectrum to obtain synthesized speech.

[0181] It should be noted that for the functions or steps that can be achieved by the above computer-readable storage medium or computer device, reference can be made to the relevant descriptions on the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described in detail here.

[0182] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to memory, storage, database, or other media used in the various embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0183] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0184] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. If software tools or components of other companies appear in the application embodiments, they are only used for example introduction and do not represent actual use. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech synthesis method based on prosody curve, characterized in that: include: Extracting text from the original speech sample and the speech signal corresponding to the text; Convert the text into a phoneme sequence, and perform duration analysis on the phoneme sequence to obtain the speech duration corresponding to each phoneme; Extracting the prosody curve of the speech signal, and performing prosody analysis on the phoneme sequence, speech duration and prosody curve to obtain prosody features corresponding to the speech signal; Acquire a latent variable, and perform mel spectrum analysis on the rhythmic feature and the latent variable to generate a mel spectrum; Perform loss calculation on the generated Mel spectrum and the preset test Mel spectrum to optimize the Mel spectrum; The pre-acquired sample text is subjected to speech synthesis with the optimized mel spectrum to obtain synthesized speech.

2. The method for speech synthesis based on prosody curve according to claim 1, characterized in that: The extracting of text from the original voice sample and the voice signal corresponding to the text includes: Converting speech content in the original speech sample into initial text; Dividing the initial text into a plurality of text segments according to preset conditions; The text segments are audio aligned according to the original speech samples to obtain a speech signal corresponding to each text segment.

3. The speech synthesis method based on prosody curve according to claim 1, characterized in that: The performing time analysis on the phoneme sequence to obtain the speech time corresponding to each phoneme includes: Performing noise reduction processing on the phoneme sequence one by one to obtain noise-reduced speech of each phoneme; Dividing the noise reduction speech into a plurality of groups of frame segment speech; Calculating the duration features of the frame segments of speech one by one to obtain the duration features corresponding to the frame segments of speech; The speech duration corresponding to each phoneme is generated according to the duration feature combination corresponding to the speech of the frame segment.

4. The method for speech synthesis based on prosody curve according to claim 1, characterized in that: The extracting the prosody curve of the speech signal comprises: Converting the speech signal into a time domain signal, and performing Fourier transform on the time domain signal to obtain a frequency domain signal; Performing pitch curve calculation on the frequency domain signal to obtain a pitch curve; Calculate the short-time energy of the speech signal and obtain the sound intensity curve; A prosody curve is generated according to the combination of the pitch curve and the intensity curve.

5. The method for speech synthesis based on prosody curve according to claim 1, characterized in that: The obtaining of latent variables comprises: Perform feature convolution on the pre-acquired mel spectrum sample data to obtain convolved mel spectrum data; Performing feature pooling on the convolutional mel spectrum data to obtain pooled mel spectrum data; Performing mean vector mapping on the pooled mel spectrum data according to a preset latent space to obtain a mean vector of the latent space distribution; Performing logarithmic variance vector mapping on the pooled mel spectrum data according to a preset latent space to obtain a logarithmic variance vector of the latent space distribution; Performing linear programming on the logarithmic variance vector and a preset random vector to obtain a programming vector; The mean vector is combined with the planning vector to generate a latent variable.

6. The method for speech synthesis based on prosody curve according to claim 1, characterized in that: The step of performing mel spectrum analysis on the prosodic feature and the latent variable to generate a mel spectrum includes: Performing feature fusion on the rhythmic feature and the latent variable to obtain a fusion result; The fusion result is mapped to the pre-acquired Mel spectrum space to obtain the Mel spectrum.

7. The method for speech synthesis based on prosody curve according to claim 1, characterized in that: The step of performing loss calculation on the generated mel spectrum and the preset test mel spectrum to optimize the mel spectrum includes: Calculating a loss value between the mel spectrum and a preset test mel spectrum; The mel spectrum is adjusted according to the loss value to obtain an optimized mel spectrum.

8. A speech synthesis device based on prosody curve, characterized in that: include: An extraction module, used to extract text from the original speech sample and the speech signal corresponding to the text; A conversion module, used for converting the text into a phoneme sequence; A duration analysis module, used to perform duration analysis on the phoneme sequence to obtain the speech duration corresponding to each phoneme; A rhythm curve extraction module, used for extracting the rhythm curve of the speech signal; A prosody analysis module, used for performing prosody analysis on the phoneme sequence, speech duration and prosody curve to obtain prosody features corresponding to the speech signal; Acquisition module, used to obtain latent variables; A mel spectrum analysis module, used for performing mel spectrum analysis on the rhythmic features and the latent variables to generate a mel spectrum; A loss calculation module, used to perform loss calculation on the generated Mel spectrum and the preset test Mel spectrum to optimize the Mel spectrum; The speech synthesis module is used to perform speech synthesis on the pre-acquired sample text and the optimized Mel spectrum to obtain synthesized speech.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the speech synthesis method based on prosody curve as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the speech synthesis method based on prosody curve as claimed in any one of claims 1 to 7 are implemented.

Images