Acoustic model training method, speech synthesis method, device and computer equipment

Abstract

The application relates to an acoustic model training method, a speech synthesis method and device and computer equipment, and is applied to the field of speech synthesis, and can improve the accuracy of text synthesized speech. The method comprises the following steps: obtaining a sample phoneme sequence, a standard mel spectrum corresponding to the sample phoneme sequence and the length of the standard mel spectrum; inputting the sample phoneme sequence and the length of the standard mel spectrum into an acoustic model to obtain a mel spectrum corresponding to the sample phoneme sequence, wherein the acoustic model comprises a phoneme embedding layer, an encoder, a length predictor, a length adjuster and a decoder; the length predictor is used for obtaining the length proportion of each phoneme based on an intermediate vector, and determining the length of the mel spectrum corresponding to each phoneme based on the length of the standard mel spectrum and the length proportion of each phoneme; and based on the mel spectrum corresponding to the sample phoneme sequence and the standard mel spectrum, the acoustic model is trained to obtain a trained target acoustic model.

Description

Technical Field

[0001] This application relates to the field of speech synthesis. More specifically, it relates to an acoustic model training method, a speech synthesis method, an apparatus, and a computer device. Background Technology

[0002] Mainstream speech synthesis backend algorithms consist of two main parts: an end-to-end speech synthesis model (also known as an acoustic model) and a vocoder. The acoustic model converts the phonemes output from the front end into Mel spectra; the vocoder converts the Mel spectra into speech.

[0003] Currently, non-autoregressive acoustic models require Automatic Speech Recognition (ASR) technology during the training phase to align phonemes with Mel spectra (referred to as the ASR alignment method) in order to obtain the duration of each phoneme in the text. However, acoustic models based on the ASR alignment method usually require separate training of the ASR alignment model, which is a complex training process. Furthermore, the alignment effect of phonemes with Mel spectra in the acoustic model is affected by the accuracy of the ASR alignment model. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, embodiments of this application provide an acoustic model training method, a speech synthesis method, an apparatus, and a computer device, which can improve the accuracy of text-to-speech synthesis.

[0005] In a first aspect, embodiments of this application provide an acoustic model training method, including:

[0006] Obtain the sample phoneme sequence, the corresponding standard Mel spectrum, and the length of the standard Mel spectrum;

[0007] The lengths of the sample phoneme sequence and the standard Mel spectrum are input into the acoustic model to obtain the Mel spectrum corresponding to the sample phoneme sequence. The acoustic model includes a phoneme embedding layer, an encoder, a length predictor, a length adjuster, and a decoder. The phoneme embedding layer is used to obtain the phoneme vectors corresponding to the sample phoneme sequence. The encoder is used to encode the phoneme vectors to obtain intermediate vectors. The length predictor is used to obtain the length ratio of each phoneme based on the intermediate vectors, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme. The length ratio is the ratio of the length of the Mel spectrum corresponding to a phoneme to the length of the standard Mel spectrum. The length adjuster is used to determine the intermediate sequence based on the intermediate vectors and the lengths of the Mel spectra corresponding to each phoneme. The decoder is used to decode the intermediate sequence to obtain the Mel spectrum corresponding to the sample phoneme sequence. The length of the intermediate sequence is the same as the length of the standard Mel spectrum.

[0008] Based on the Mel spectrum corresponding to the sample phoneme sequence and the standard Mel spectrum, the acoustic model is trained to obtain the trained target acoustic model.

[0009] Secondly, embodiments of this application provide a speech synthesis method, the method comprising:

[0010] Obtain the text to be synthesized speech;

[0011] Based on the length of the speech text to be synthesized, determine the length of the target Mel spectrum corresponding to the speech text to be synthesized;

[0012] Convert the speech text to be synthesized into a target phoneme sequence;

[0013] The target phoneme sequence and the length of the target Mel spectrum are input into the target acoustic model to obtain the target Mel spectrum. The target acoustic model is trained using the acoustic model training method described in the first aspect.

[0014] The target speech is generated from the Mel spectrum of the target speech text using a vocoder.

[0015] Thirdly, embodiments of this application provide an acoustic model training device, including:

[0016] The acquisition module is used to acquire the sample phoneme sequence, the standard Mel spectrum corresponding to the sample phoneme sequence, and the length of the standard Mel spectrum;

[0017] The input module is used to input the lengths of the sample phoneme sequence and the standard Mel spectrum into the acoustic model to obtain the Mel spectrum corresponding to the sample phoneme sequence. The acoustic model includes a phoneme embedding layer, an encoder, a length predictor, a length adjuster, and a decoder. The phoneme embedding layer is used to obtain the phoneme vectors corresponding to the sample phoneme sequence. The encoder is used to encode the phoneme vectors to obtain intermediate vectors. The length predictor is used to obtain the length ratio of each phoneme based on the intermediate vectors, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme. The length ratio is the ratio of the length of the Mel spectrum corresponding to a phoneme to the length of the standard Mel spectrum. The length adjuster is used to determine the intermediate sequence based on the intermediate vectors and the lengths of the Mel spectra corresponding to each phoneme. The decoder is used to decode the intermediate sequence to obtain the Mel spectrum corresponding to the sample phoneme sequence. The length of the intermediate sequence is the same as the length of the standard Mel spectrum.

[0018] The training module is used to train the acoustic model based on the Mel spectrum corresponding to the sample phoneme sequence and the standard Mel spectrum, so as to obtain the trained target acoustic model.

[0019] Fourthly, embodiments of this application provide a speech synthesis device, including:

[0020] The acquisition module is used to acquire the text to be synthesized speech;

[0021] The determining module is used to determine the length of the target Mel spectrum corresponding to the speech text to be synthesized based on the text length of the speech text to be synthesized;

[0022] The conversion module is used to convert the speech text to be synthesized into a target phoneme sequence;

[0023] An input module is used to input the target phoneme sequence and the length of the target Mel spectrum into the target acoustic model to obtain the target Mel spectrum, which is trained by the acoustic model training method described in the first aspect;

[0024] The generation module is used to generate the target speech corresponding to the speech text to be synthesized from the target Mel spectrum using a vocoder.

[0025] Fifthly, embodiments of this application provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the acoustic model training method as described in the first aspect or the speech synthesis method as described in the second aspect.

[0026] In a sixth aspect, embodiments of this application provide a computer-readable storage medium, including: storing a computer program on the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the acoustic model training method or the speech synthesis method as described in the second aspect.

[0027] In a seventh aspect, embodiments of this application provide a computer program product, including: when the computer program product is run on a computer, causing the computer to implement the acoustic model training method or the speech synthesis method as described in the second aspect.

[0028] Compared with the prior art, the technical solution provided in this application has the following advantages: In this application embodiment, a sample phoneme sequence, the standard Mel spectrum corresponding to the sample phoneme sequence, and the length of the standard Mel spectrum are obtained; the sample phoneme sequence and the length of the standard Mel spectrum are input into an acoustic model to obtain the Mel spectrum corresponding to the sample phoneme sequence. The acoustic model includes a phoneme embedding layer, an encoder, a length predictor, a length adjuster, and a decoder; the phoneme embedding layer is used to obtain the phoneme vector corresponding to the sample phoneme sequence; the encoder is used to encode the phoneme vector to obtain an intermediate vector; the length predictor is used to obtain the length ratio corresponding to each phoneme based on the intermediate vector, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme, where the length ratio is the ratio of the length of the Mel spectrum corresponding to a phoneme to the length of the standard Mel spectrum; the length adjuster is used to determine the intermediate sequence based on the intermediate vector and the length of the Mel spectrum corresponding to each phoneme; the decoder is used to decode the intermediate sequence to obtain the Mel spectrum corresponding to the sample phoneme sequence, where the length of the intermediate sequence is the same as the length of the standard Mel spectrum; the acoustic model is trained based on the Mel spectrum corresponding to the sample phoneme sequence and the standard Mel spectrum to obtain a trained target acoustic model. Therefore, a length predictor is added to the acoustic model to obtain the length proportion of each phoneme based on the intermediate vector, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length proportion of each phoneme. This allows the standard Mel spectrum to be used to predict the duration of phonemes during the acoustic model training process, thereby achieving alignment between phonemes and Mel spectra and ensuring that the length of the intermediate sequence is the same as the length of the standard Mel spectrum. This guarantees the accuracy of the conversion from phoneme sequence to Mel spectrum. The proposed solution does not require the use of an ASR alignment model for alignment, simplifying the acoustic model training process and avoiding the impact of the accuracy of the ASR alignment model on the acoustic model's performance. Attached Figure Description

[0029] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0030] Figure 1 A scenario architecture diagram of a speech synthesis method according to some embodiments is shown;

[0031] Figure 2 A hardware configuration block diagram of an electronic device according to some embodiments is shown;

[0032] Figure 3 A schematic diagram of the operating system of an electronic device and a server according to some embodiments is shown;

[0033] Figure 4 A flowchart illustrating an acoustic model training method according to some embodiments is shown;

[0034] Figure 5 One of the structural schematic diagrams of an acoustic model according to some embodiments is shown;

[0035] Figure 6 A second schematic diagram of the structure of an acoustic model according to some embodiments is shown;

[0036] Figure 7 The third schematic diagram of the acoustic model according to some embodiments is shown;

[0037] Figure 8 The fourth schematic diagram of the structure of an acoustic model according to some embodiments is shown;

[0038] Figure 9 Fifth of the schematic diagrams of the acoustic model according to some embodiments is shown;

[0039] Figure 10 One of the flowcharts of a speech synthesis method according to some embodiments is shown;

[0040] Figure 11 A second schematic flowchart of a speech synthesis method according to some embodiments is shown;

[0041] Figure 12 A schematic diagram of the framework of an acoustic model training apparatus according to some embodiments is shown;

[0042] Figure 13 A schematic diagram of the framework of a speech synthesis device according to some embodiments is shown;

[0043] Figure 14 A schematic diagram of computer device hardware according to some embodiments is shown. Detailed Implementation

[0044] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0045] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0046] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0047] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0048] Figure 1 This is a schematic diagram of a scenario architecture for a speech synthesis method provided in an embodiment of this application. Figure 1 As shown in the embodiment of this application, the scenario architecture includes: a server 100 and an electronic device 200.

[0049] The electronic device 200 provided in this application embodiment can have various implementation forms, such as smart speakers, televisions, refrigerators, washing machines, air conditioners, smart curtains, routers, set-top boxes, mobile phones, personal computers (PCs), smart TVs, laser projection equipment, monitors, electronic bulletin boards, wearable devices, in-vehicle equipment, electronic tables, etc.

[0050] In some embodiments, when the electronic device 200 receives an instruction to synthesize text into speech, the electronic device 200 can implement the text-to-speech function itself, or it can implement the text-to-speech function through data communication with the server 100. The electronic device 200 may communicate with the server 100 via a local area network (LAN) or a wireless local area network (WLAN).

[0051] Server 100 can be a server that provides various services, such as a server that provides speech synthesis support for text acquired by electronic device 200. The server can perform synthesis processing on the received text data and feed back the processing result (e.g., synthesized speech) to electronic device 200. Server 100 can be a server cluster or multiple server clusters, and can include one or more types of servers.

[0052] Electronic device 200 can be either hardware or software. When electronic device 200 is hardware, it can be various electronic devices with sound playback capabilities, including but not limited to smart speakers, smartphones, televisions, tablets, e-book readers, smartwatches, media players, computers, AI devices, robots, smart vehicles, etc. When electronic device 200 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules (e.g., to provide speech synthesis services) or as a single software program or software module. No specific limitations are made here.

[0053] For example, electronic device 200 receives a user's instruction to convert text into speech, and then sends the text to be synthesized to server 100. Server 100 processes the text to be synthesized using the speech synthesis method provided in this application embodiment to obtain the target speech corresponding to the text to be synthesized, and then returns the target speech to electronic device 200, which then broadcasts the target speech.

[0054] It should be noted that, Figure 1 The schematic diagram shown illustrates only one possible scenario for implementing the speech synthesis method provided in this embodiment. The execution entity of the speech synthesis method provided in this application embodiment can be the aforementioned server or a server containing a functional module or entity that implements the speech synthesis method. The execution entity of the speech synthesis method provided in this application embodiment can also be the aforementioned electronic device or an electronic device containing a functional module or entity that implements the speech synthesis method.

[0055] Figure 2 A hardware configuration block diagram of an electronic device 200 according to an exemplary embodiment is shown. For example... Figure 2 The illustrated electronic device 200 includes at least one of the following: a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, an audio processor, RAM, ROM, and a first to an nth interface for input / output.

[0056] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The electronic device 200 can establish the transmission and reception of control signals and data signals through the communicator 220 and server 100.

[0057] User interface 280 can be used to receive external control signals.

[0058] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0059] A sound acquisition device can be a microphone, also known as a "microphone" or "voice transducer," which can be used to receive the user's voice and convert the sound signal into an electrical signal. Electronic device 200 can be equipped with at least one microphone. In some embodiments, electronic device 200 can be equipped with two microphones, which, in addition to acquiring sound signals, can also perform noise reduction. In other embodiments, electronic device 200 can also be equipped with three, four, or more microphones, enabling sound signal acquisition, noise reduction, sound source identification, and directional recording functions, etc.

[0060] Furthermore, the microphone may be built into the electronic device 200, or it may be connected to the electronic device 200 via wired or wireless means. Of course, this application embodiment does not limit the location of the microphone on the electronic device 200. Alternatively, the electronic device 200 may not include a microphone, meaning the microphone is not located within the electronic device 200. The electronic device 200 can connect an external microphone (also called a microphone) via an interface (such as a USB interface 130). This external microphone can be fixed to the electronic device 200 using an external fastener (such as a camera bracket with a clip).

[0061] The controller 250 controls the operation of the electronic device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the electronic device 200.

[0062] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0063] In some examples, the electronic device uses the Android operating system, such as... Figure 3 As shown, the electronic device 200 can be logically divided into an application layer (referred to as "application layer") 21, a kernel layer 22, and a hardware layer 23.

[0064] Among them, such as Figure 3 As shown, hardware layer 23 may include Figure 2 The controller 250, communicator 220, detector 230, etc., are shown. Application layer 21 includes one or more applications. These applications can be system applications or third-party applications. For example, application layer 21 includes a voice recognition application, which can provide a voice interaction interface and services for connecting electronic device 200 to server 100.

[0065] The kernel layer 22 serves as a software middleware between the hardware layer 23 and the application layer 21, and is used to manage and control hardware and software resources.

[0066] like Figure 3 As shown, the server 100 may include a communication control module 101, a speech synthesis module 102, and a data storage module 103, and may also include other modules, which are not limited here. The communication control module 101 is used to communicate with the communicator 220, and the data storage module 103 is used to store various databases; in this embodiment, it can be used to store tabular data.

[0067] In some examples, kernel layer 22 includes a detector driver. When the speech synthesis application in electronic device 200 is started and a communication connection is established between electronic device 200 and server 100, the detector driver sends the speech text to be synthesized collected by detector 230 to the speech synthesis application. The speech synthesis application then sends the speech text to be synthesized to the speech synthesis module 102 in the server. The speech synthesis module 102 takes the speech text sent by electronic device 200 as input to the speech synthesis model, obtains the speech synthesis result, and then transmits the speech synthesis result back to electronic device 200.

[0068] Traditional Text-to-Speech (TTS) models mainly consist of two parts: front-end text-to-phoneme conversion and back-end phoneme-to-speech signal conversion. The front-end converts the initial text into phonemes, while the back-end upsamples the phonemes into a speech signal. Building these modules requires a large amount of prior knowledge and significant time for feature design. With the development of deep learning in the speech domain, end-to-end speech conversion schemes based on deep neural networks have gradually emerged. Currently, the back-end uses acoustic features as a transition, essentially achieving end-to-end training and synthesis.

[0069] Text front-end processing generally involves three basic steps: text preprocessing, text normalization, and phoneme analysis. It also predicts prosody from the text and normalized text. Phoneme and prosodic identifiers are collectively referred to as linguistic features. The output of the text front-end serves as the input to the downstream acoustic model and vocoder. If pronunciation errors occur, in most cases, the phoneme sequence can be directly corrected, greatly reducing the difficulty of problem-solving.

[0070] The backend typically consists of two parts: phoneme-acoustic feature (Mel spectrum) conversion (acoustic model) and acoustic feature-speech signal conversion (vocoder).

[0071] The reason for dividing the backend into two segments is that TTS involves upsampling. Speech signals have a high sampling rate, typically 16kHz, with each word corresponding to many audio sampling points. To achieve good results, it is generally necessary to use acoustic features as a transition, performing upsampling in two steps. Mel spectrum is the most common acoustic feature for this transition.

[0072] Currently, vocoders can be broadly categorized into phase reconstruction-based vocoders and neural network-based vocoders. Phase reconstruction-based vocoders address the issue that the acoustic features (MEL features, etc.) used in TTS have lost phase characteristics; therefore, algorithms are used to infer the phase features and reconstruct the speech waveform. Neural network-based vocoders, on the other hand, directly map acoustic features to the speech waveform, resulting in higher synthesized sound quality. Popular neural network vocoders include WaveNet, WaveRNN, Melgan, Waveglow, and FastSpeech. This application focuses on the phoneme-MEL spectrum conversion part, which is a core component of TTS and plays a decisive role in the synthesis effect.

[0073] The phoneme-Mel spectrum conversion part is mainly divided into autoregressive models and non-autoregressive models. Autoregressive models, such as tacotron2 and transformerTTS, do not require phoneme alignment information; the model learns which phoneme corresponds to each step and then decides whether to stop. These types of autoregressive models must execute step by step, ending when the maximum number of steps is determined automatically or manually, resulting in very slow synthesis speeds.

[0074] Non-autoregressive models are significantly faster than autoregressive models. Non-autoregressive models, such as FastSpeech, offer better synthesis results and faster speed, but training requires an ASR alignment model to obtain the phoneme-to-mel spectrum alignment (or simply phoneme alignment). Generally, for the same ten words, the rate of generating a mel spectrum is in the tens of milliseconds for FastSpeech and in the hundreds of milliseconds for Tacotron2. Therefore, the synthesis speed advantage of non-autoregressive models can be demonstrated, but the phoneme alignment problem remains to be solved.

[0075] Speech synthesis is an "upsampling" process, where the length of acoustic features such as the Mel spectrum is often much greater than the number of phonemes. Furthermore, speech is random, with the length of each phoneme's pronunciation varying considerably. Therefore, both training and inference require determining the length of each phoneme and finding its corresponding segment in the Mel spectrum, a process known as "alignment."

[0076] End-to-end speech synthesis models use a trained acoustic model to generate Mel spectra from the phonemes output from the front end, and then convert the Mel spectra into speech signals (synthesized speech) using a vocoder. Non-autoregressive end-to-end models require the actual duration of each phoneme in the audio during the training phase. This is typically achieved using an ASR alignment model for phoneme training. For example, the mainstream algorithm FastSpeech relies on the phoneme alignment results of the ASR alignment model during training, which presents the following problems: 1) The training process is complex, and a separate ASR alignment model is generally required to achieve good results; 2) The phoneme alignment effect is affected by the accuracy of the ASR alignment model; 3) There are rounding errors in the estimation of phoneme frame counts.

[0077] Therefore, existing non-autoregressive end-to-end models face several challenges during the training phase. On the one hand, the training process is complex because an ASR alignment model needs to be trained separately. On the other hand, the phoneme alignment achieved through the ASR alignment model may contain subtle errors, which can lead to errors in the generated Mel spectrum and ultimately result in imperfections in the synthesized speech.

[0078] To address the aforementioned technical problems, in some embodiments of this application, a sample phoneme sequence, the corresponding standard Mel spectrum, and the length of the standard Mel spectrum are obtained. The sample phoneme sequence and the length of the standard Mel spectrum are input into an acoustic model to obtain the Mel spectrum corresponding to the sample phoneme sequence. The acoustic model includes a phoneme embedding layer, an encoder, a length predictor, a length adjuster, and a decoder. The phoneme embedding layer is used to obtain the phoneme vector corresponding to the sample phoneme sequence. The encoder is used to encode the phoneme vector to obtain an intermediate vector. The length predictor is used to obtain the length ratio of each phoneme based on the intermediate vector, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme. The length ratio is the ratio of the length of the Mel spectrum corresponding to a phoneme to the length of the standard Mel spectrum. The length adjuster is used to determine the intermediate sequence based on the intermediate vector and the length of the Mel spectrum corresponding to each phoneme. The decoder is used to decode the intermediate sequence to obtain the Mel spectrum corresponding to the sample phoneme sequence. The length of the intermediate sequence is the same as the length of the standard Mel spectrum. Based on the Mel spectrum corresponding to the sample phoneme sequence and the standard Mel spectrum, the acoustic model is trained to obtain a trained target acoustic model. Therefore, a length predictor is added to the acoustic model to obtain the length proportion of each phoneme based on the intermediate vector, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length proportion of each phoneme. This allows the standard Mel spectrum to be used to predict the duration of phonemes during the acoustic model training process, thereby achieving alignment between phonemes and Mel spectra and ensuring that the length of the intermediate sequence is the same as the length of the standard Mel spectrum. This guarantees the accuracy of the conversion from phoneme sequence to Mel spectrum. The proposed solution does not require the use of an ASR alignment model for alignment, simplifying the acoustic model training process, improving the accuracy of phoneme alignment, and avoiding the impact of the accuracy of the ASR alignment model on the acoustic model's performance. This improves the training process of the TTS model, avoids the accumulation of errors, achieves stable synthesis results, and enhances the overall synthesis effect.

[0079] Figure 4 This is a flowchart illustrating the steps of implementing an acoustic model training method according to one or more embodiments of this application. The executing entity of the acoustic model training method can be a server or an electronic device, or a functional module or entity within the server or electronic device capable of implementing the acoustic model training method; no limitation is made here. In the embodiments of this application, the acoustic model training method may include the following steps S401 to S403.

[0080] S401. Obtain the sample phoneme sequence, the standard Mel spectrum corresponding to the sample phoneme sequence, and the length of the standard Mel spectrum.

[0081] S402. Input the lengths of the sample phoneme sequence and the standard Mel spectrum into the acoustic model to obtain the Mel spectrum corresponding to the sample phoneme sequence.

[0082] Among them, such as Figure 5 As shown, the acoustic model 50 includes a phoneme embedding layer 51, an encoder 52, a length predictor 53, a length adjuster 54, and a decoder 55. The phoneme embedding layer 51 is used to obtain the phoneme vectors corresponding to the sample phoneme sequence. The encoder 52 is used to encode the phoneme vectors to obtain intermediate vectors. The length predictor 53 is used to obtain the length ratio of each phoneme based on the intermediate vectors, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme. The length adjuster 54 is used to determine the intermediate sequence based on the intermediate vectors and the length of the Mel spectrum corresponding to each phoneme. The decoder 55 is used to decode the intermediate sequence to obtain the Mel spectrum corresponding to the sample phoneme sequence. The length of the intermediate sequence is the same as the length of the standard Mel spectrum.

[0083] Among them, the intermediate vector can be the prosodic latent feature.

[0084] Here, the length percentage is the ratio of the length of the Mel spectrum corresponding to a phoneme to the length of the standard Mel spectrum. Specifically, the length predictor predicts the length of the Mel spectrum corresponding to each phoneme based on the intermediate vector, then normalizes the length of the Mel spectrum corresponding to each phoneme to obtain the length percentage for each phoneme, and finally determines the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length percentage of each phoneme.

[0085] In some embodiments of this application, the length predictor may include a phoneme length prediction module (PortionPredictor), a normalization module (e.g., softmax), and a product module. The phoneme length prediction module is used to predict the length of the Mel spectrum corresponding to each phoneme based on the intermediate vector. The normalization module is used to normalize the length of the Mel spectrum corresponding to each phoneme to obtain the length ratio of each phoneme. The product module is used to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme.

[0086] Specifically, the phoneme length prediction module is used to predict the length of the Mel spectrum corresponding to each phoneme based on the intermediate vector and the standard Mel spectrum.

[0087] The main structure of the length predictor can be a convolutional network, but the specific network structure is not limited in the embodiments of this application.

[0088] In some embodiments of this application, the length can be a time length (in seconds) or an integer multiple of a fixed time length (in units), which can be determined according to the actual situation and is not limited here. Among them, the fixed time length can be determined according to the actual situation, for example, it can be the time length occupied by one phoneme.

[0089] Among them, the length of the middle sequence is the same as that of the standard Mel spectrum. Therefore, decoding the middle sequence results in a Mel spectrum corresponding to the sample phoneme sequence with the same length as the standard Mel spectrum, and thus the Mel spectrum corresponding to the sample phoneme sequence is closer to the standard Mel spectrum.

[0090] Among them, the network structure of the decoder can be the same as or different from that of the encoder, which is not limited here.

[0091] In some embodiments of the present application, there are many Chinese characters, and it is difficult for training samples to cover them. Therefore, Chinese characters are usually converted into phonemes, and speech is synthesized through phonemes. For example, when synthesizing the complete speech information "Hello" in a TV scenario, the text is split into a phoneme sequence "eos n i2 h aa3 uu3 eos". Among them, "eos" is the endpoint.

[0092] In an embodiment of the present application, the sample phoneme sequence and the standard Mel spectrum are input into the acoustic model. The phoneme embedding layer embeds the sample phoneme sequence into the vector space to obtain the corresponding phoneme vector, and the phoneme vector is denoted as x l , where l∈[0,1,…,L] represents the position of the phoneme from left to right, and L is the number of phonemes. Then x l is passed into the encoder to generate an intermediate vector h l . The encoder fuses the relationship between the front and back of the phonemes and can model the changes generated by the phonemes in the context. The encoder generally uses N feed-forward Transformer blocks (FFT Blocks), and it is better to take N as 4-6. Since the total length of the standard Mel spectrum is known, the length of the Mel spectrum corresponding to each phoneme can be determined by predicting the proportion of each phoneme. The intermediate vector h l is passed into the phoneme length prediction module, and the result is normalized through the Softmax layer to obtain the final proportion p of each phoneme l , where Since the speech is given during training, the length of the standard Mel spectrum is known. Denote the length of the standard Mel spectrum as T, then T l =T*p l is the length of the Mel spectrum corresponding to each phoneme. h l ,T l is passed into the length regulator to obtain an intermediate sequence of the same length as the standard Mel spectrum. Finally, the intermediate sequence is passed into the decoder to obtain the Mel spectrum.

[0093] S403. Based on the Mel spectrum corresponding to the sample phoneme sequence and the standard Mel spectrum, train the acoustic model to obtain a trained target acoustic model.

[0094] In this embodiment, a length predictor is added to the acoustic model to obtain the length proportion of each phoneme based on the intermediate vector, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length proportion of each phoneme. This allows the standard Mel spectrum to be used to predict the duration of phonemes during acoustic model training, achieving alignment between phonemes and the Mel spectrum, and ensuring that the length of the intermediate sequence is the same as the length of the standard Mel spectrum. This guarantees the accuracy of the phoneme sequence to Mel spectrum conversion. This solution eliminates the need for an ASR alignment model, simplifying the acoustic model training process and avoiding the impact of ASR alignment model accuracy on the acoustic model's performance. This improves the accuracy of phoneme alignment and avoids the impact of ASR alignment model accuracy on the acoustic model's performance, thereby improving the TTS model training process, preventing error accumulation, achieving stable synthesis results, and enhancing the overall synthesis effect.

[0095] In some embodiments of this application, the length regulator is a length regulator that adjusts the phonemes according to their respective durations (T). l The sequence is copied to obtain the copied sequence corresponding to each phoneme. Then, the copied sequences corresponding to each phoneme are spliced ​​together to obtain an intermediate sequence with the same length as the standard Mel spectrum.

[0096] In some embodiments of this application, combined with Figure 5 ,like Figure 6 As shown, the acoustic model 50 also includes: a standard deviation predictor 56; the standard deviation predictor 56 is used to obtain the standard deviation corresponding to each phoneme based on the intermediate vector; the length adjuster 54 is specifically a soft length adjuster based on the Gaussian probability density function corresponding to each phoneme, the Gaussian probability density function corresponding to a phoneme is determined by the standard deviation corresponding to the phoneme and the center position corresponding to the phoneme, and the center position corresponding to the phoneme is determined according to the length of the Mel spectrum corresponding to each phoneme.

[0097] In Mel-spectrum time-frequency analysis, phonemes often lack clear boundaries, with partial overlap between them. Furthermore, continuous variations occur between phonemes during articulation. To model these two phenomena, a Standard Deviation Predictor (STD Predictor) is introduced to model overlap and continuous variations. The intermediate vector h is then used... l The standard deviation parameter σ is obtained by inputting the standard deviation predictor and taking the absolute value (ABS) of the result. l The corresponding h l ,σ l ,T l , pass in Soft Length Regulator.

[0098] The specific implementation steps of the soft length regulator in this embodiment are as follows:

[0099] Step 1: Obtain the center position μ for each phoneme l ,

[0100] Step 2: Using μ l and variance σ l The mean of the construction is μ l The variance is σ l Gaussian probability density function D l (t), where t∈[0,1,…,T], the function takes the maximum value at the center of the phoneme and gradually decreases towards both sides;

[0101] Step 3: Encode the phonemes containing fundamental frequency and energy information in the middle (h) l Copy T times to get The length is consistent with the Mel spectrum;

[0102] Step 4: Use D l (t) Emphasize the central position of the phoneme to obtain the final code H of the phoneme. l ,Right now

[0103] Step 5: Encode each phoneme H l Summing yields the complete statement's encoding H, i.e.

[0104] In summary, this embodiment combines a standard deviation predictor and a soft length modulator to model the overlap and continuous changes between phonemes. It also utilizes the characteristics of the Gaussian probability density function to highlight intermediate phonemes and employs a gentler approach to the boundaries. The resulting intermediate sequences more closely resemble the true standard Mel spectrum, improving the accuracy of the phoneme sequence-to-Mel spectrum conversion and thus enhancing the accuracy of speech synthesis.

[0105] In some embodiments of this application, combined with Figure 6 ,like Figure 7 As shown, the acoustic model 50 further includes: a fundamental frequency predictor 571 and a fundamental frequency embedding layer 572; the fundamental frequency predictor 571 is used to determine the fundamental frequency parameter sequence based on the intermediate vector, and the fundamental frequency embedding layer 572 is used to obtain the fundamental frequency parameter vector corresponding to the fundamental frequency parameter sequence; the length adjuster 54 is specifically used to determine the intermediate sequence based on the first vector and the length of the Mel spectrum corresponding to each phoneme, wherein the first vector is the sum of the intermediate vector and the fundamental frequency parameter vector.

[0106] In this embodiment, to enrich the prosody of the speech and avoid overly flat intonation, an independent fundamental frequency predictor is constructed after the encoder to model the pitch variations. Encoding the pitch corresponding to each phoneme allows the Mel spectrum obtained from the phoneme sequence conversion to be closer to the standard Mel spectrum, resulting in more accurate acoustic features and thus improving the accuracy of speech synthesis. The intermediate vector h is then used... l The input is passed to the base frequency predictor (F0 Predictor) to obtain the base frequency parameter f. l Then f l The signal is passed into the fundamental frequency embedding layer (F0-Embedding) to obtain the final embedding result and h. l Adding them together gives q l The corresponding q l ,σ l ,T l The input is the Soft Length Regulator. Step 3 in the soft length regulator is to set q... l Copy T times to get

[0107] In some embodiments of this application, combined with Figure 6 ,like Figure 8 As shown, the acoustic model 50 further includes: an energy predictor 581 and an energy embedding layer 582; the energy predictor 581 is used to determine the energy parameter sequence based on the intermediate vector, and the energy embedding layer 582 is used to obtain the energy parameter vector corresponding to the energy parameter sequence; the length adjuster 54 is specifically used to determine the intermediate sequence based on the second vector and the length of the Mel spectrum corresponding to each phoneme, wherein the second vector is the sum of the intermediate vector and the energy parameter vector.

[0108] In this embodiment, an independent energy predictor is constructed after the encoder to model the intensity variations of speech. Encoding the intensity variations of speech within phonemes allows the Mel spectrum obtained from phoneme sequence conversion to more closely approximate the standard Mel spectrum, resulting in more accurate acoustic features and thus improving the accuracy of speech synthesis. The intermediate vector h is then used to... l Input to the Energy Predictor to obtain the energy parameter g. l Then g l Input Energy-Embedding to obtain the final embedding result and h l Add them together to get w l The corresponding w l ,σ l ,T l The input is the Soft Length Regulator. Step 3 in the soft length regulator is to adjust w. l Copy T times to get

[0109] In some embodiments of this application, combined with Figure 8 ,like Figure 9 As shown, the acoustic model further includes: a fundamental frequency predictor 571 and a fundamental frequency embedding layer 572; the fundamental frequency predictor 571 is used to determine the fundamental frequency parameter sequence based on the intermediate vector, and the fundamental frequency embedding layer 572 is used to obtain the fundamental frequency parameter vector corresponding to the fundamental frequency parameter sequence; the length adjuster 54 is specifically used to determine the intermediate sequence based on the third vector and the length of the Mel spectrum corresponding to each phoneme, wherein the third vector is the sum of the intermediate vector, the fundamental frequency parameter vector and the energy parameter vector.

[0110] In this embodiment, to enrich the prosody of the speech and avoid overly flat intonation, an independent fundamental frequency predictor is constructed after the encoder to model the pitch variations. Encoding the pitch corresponding to each phoneme allows the Mel spectrum obtained from the phoneme sequence conversion to be closer to the standard Mel spectrum, resulting in more accurate acoustic features and thus improving the accuracy of speech synthesis. The intermediate vector h is then used... l The input is passed to the base frequency predictor (F0 Predictor) to obtain the base frequency parameter f. l Then f l The signal is passed into the fundamental frequency embedding layer (F0-Embedding) to obtain the final embedding result and w. l Adding them together gives s l The corresponding s l ,σ l ,T l The input is the Soft Length Regulator. Step 3 in the soft length regulator is to adjust s... l Copy T times to get

[0111] In some embodiments of this application, the acoustic model described above can form an end-to-end speech synthesis model with a vocoder and a front-end model. The front-end model is used to convert sample text into a sample phoneme sequence, and the vocoder is used to convert the Mel spectrum corresponding to the sample phoneme sequence into a speech signal.

[0112] Figure 10 The flowchart illustrates the steps of implementing a speech synthesis method according to one or more embodiments of this application. The execution entity of the speech synthesis method can be a server or an electronic device, or a functional module or entity within the server or electronic device capable of implementing the speech synthesis method; no limitation is made here. In the embodiments of this application, the speech synthesis method may include the following steps S1001 to S1005.

[0113] S1001. Obtain the text to be synthesized speech.

[0114] S1002. Determine the length of the target Mel spectrum corresponding to the speech text to be synthesized based on the text length of the speech text to be synthesized.

[0115] S1003. Convert the speech text to be synthesized into a target phoneme sequence.

[0116] S1004. Input the target phoneme sequence and the length of the target Mel spectrum into the target acoustic model to obtain the target Mel spectrum.

[0117] The target acoustic model is obtained by training using the acoustic model training method described above.

[0118] S1005. The target speech corresponding to the speech text to be synthesized is generated by using the vocoder based on the target Mel spectrum.

[0119] In this embodiment of the application, the acoustic model trained by the above-mentioned acoustic model training method is used to improve the accuracy of phoneme alignment during the inference process, thereby achieving a stable synthesis effect and improving the synthesis accuracy.

[0120] In some embodiments of this application, combined with Figure 10 ,like Figure 11 As shown, the above S1002 can be specifically implemented through the following S1002a and S1002b.

[0121] S1002a. Determine the initial length of the target Mel spectrum corresponding to the speech text to be synthesized based on the text length of the speech text to be synthesized.

[0122] S1002b: Adjust the initial length of the target Mel spectrum according to the speech rate requirement of the received synthesized speech to obtain the length of the target Mel spectrum.

[0123] In addition to obtaining more accurate Mel spectrum during the model training stage, this invention can also control the speech rate of speech synthesis during the inference stage by controlling the length T of the Mel spectrum. The larger T is, the slower the speech rate, and vice versa.

[0124] Figure 12 This is a structural block diagram of the acoustic model training device shown in the embodiments of this application, such as... Figure 12 As shown, the device includes:

[0125] The acquisition module 1201 is used to acquire the sample phoneme sequence, the standard Mel spectrum corresponding to the sample phoneme sequence, and the length of the standard Mel spectrum;

[0126] Input module 1202 is used to input the lengths of the sample phoneme sequence and the standard Mel spectrum into the acoustic model to obtain the Mel spectrum corresponding to the sample phoneme sequence. The acoustic model includes a phoneme embedding layer, an encoder, a length predictor, a length adjuster, and a decoder. The phoneme embedding layer is used to obtain the phoneme vectors corresponding to the sample phoneme sequence. The encoder is used to encode the phoneme vectors to obtain intermediate vectors. The length predictor is used to obtain the length ratio of each phoneme based on the intermediate vectors, and to determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme. The length ratio is the ratio of the length of the Mel spectrum corresponding to a phoneme to the length of the standard Mel spectrum. The length adjuster is used to determine the intermediate sequence based on the intermediate vectors and the lengths of the Mel spectra corresponding to each phoneme. The decoder is used to decode the intermediate sequence to obtain the Mel spectrum corresponding to the sample phoneme sequence. The length of the intermediate sequence is the same as the length of the standard Mel spectrum.

[0127] Training module 1203 is used to train an acoustic model based on the Mel spectrum corresponding to the sample phoneme sequence and the standard Mel spectrum, so as to obtain a trained target acoustic model.

[0128] In some embodiments of this application, the acoustic model further includes: a standard deviation predictor;

[0129] This standard deviation predictor is used to obtain the standard deviation of each phoneme based on the intermediate vector;

[0130] Specifically, the length adjuster is a soft length adjuster based on the Gaussian probability density function corresponding to each phoneme. The Gaussian probability density function corresponding to a phoneme is determined by the standard deviation of that phoneme and the center position of that phoneme. The center position of that phoneme is determined according to the length of the Mel spectrum corresponding to each phoneme.

[0131] In some embodiments of this application, the acoustic model further includes: a fundamental frequency predictor and a fundamental frequency embedding layer;

[0132] The baseband predictor is used to determine the baseband parameter sequence based on the intermediate vector, and the baseband embedding layer is used to obtain the baseband parameter vector corresponding to the baseband parameter sequence;

[0133] The length adjuster is specifically used to determine the intermediate sequence based on the length of the first vector and the Mel spectrum corresponding to each phoneme, wherein the first vector is the sum of the intermediate vector and the fundamental frequency parameter vector.

[0134] In some embodiments of this application, the acoustic model further includes: an energy predictor and an energy embedding layer;

[0135] The energy predictor is used to determine the energy parameter sequence based on the intermediate vector, and the energy embedding layer is used to obtain the energy parameter vector corresponding to the energy parameter sequence.

[0136] The length adjuster is specifically used to determine the intermediate sequence based on the length of the Mel spectrum corresponding to each phoneme and the second vector, which is the sum of the intermediate vector and the energy parameter vector.

[0137] In some embodiments of this application, the acoustic model further includes: a fundamental frequency predictor and a fundamental frequency embedding layer;

[0138] The baseband predictor is used to determine the baseband parameter sequence based on the intermediate vector, and the baseband embedding layer is used to obtain the baseband parameter vector corresponding to the baseband parameter sequence;

[0139] The length adjuster is specifically used to determine the intermediate sequence based on the length of the third vector and the Mel spectrum corresponding to each phoneme. The third vector is the sum of the intermediate vector, the fundamental frequency parameter vector, and the energy parameter vector.

[0140] In this embodiment, each module of the acoustic model training device can implement the acoustic model training method provided in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0141] Figure 13 This is a structural block diagram of a speech synthesis device shown in an embodiment of this application, such as... Figure 13 As shown, the device includes:

[0142] The acquisition module 1301 is used to acquire the text to be synthesized speech;

[0143] The determining module 1302 is used to determine the length of the target Mel spectrum corresponding to the speech text to be synthesized based on the text length of the speech text to be synthesized;

[0144] The conversion module 1303 is used to convert the speech text to be synthesized into a target phoneme sequence;

[0145] The input module 1304 is used to input the target phoneme sequence and the length of the target Mel spectrum into the target acoustic model to obtain the target Mel spectrum. The target acoustic model is trained by the acoustic model training method described above.

[0146] The generation module 1305 is used to generate the target speech corresponding to the speech text to be synthesized by using the target Mel spectrum of the target speech text through a vocoder.

[0147] In some embodiments of this application, the determining module 1302 is specifically used to determine the initial length of the target Mel spectrum corresponding to the speech text to be synthesized based on the text length of the speech text to be synthesized; and to adjust the initial length of the target Mel spectrum according to the speech rate requirements of the received synthesized speech to obtain the length of the target Mel spectrum.

[0148] In this embodiment, each module of the speech synthesis device can implement the speech synthesis method provided in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0149] like Figure 14 As shown in the figure, this application embodiment also provides a computer device 1400, which can be the above-mentioned electronic device or server. The computer device 1400 includes: a processor 1401, a memory 1402, and a computer program stored in the memory 1402 and executable on the processor 1401. When the computer program is executed by the processor 1401, it implements the various processes performed by the acoustic model training method or speech synthesis method described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0150] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described acoustic model training method or speech synthesis method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0151] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0152] This application provides a computer program product, including: when the computer program product is run on a computer, causing the computer to implement the above-described acoustic model training method or speech synthesis method.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0154] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. An acoustic model training method, characterized in that, include: Obtain the sample phoneme sequence, the corresponding standard Mel spectrum, and the length of the standard Mel spectrum; The lengths of the sample phoneme sequence and the standard Mel spectrum are input into the acoustic model to obtain the Mel spectrum corresponding to the sample phoneme sequence. The acoustic model includes a phoneme embedding layer, an encoder, a length predictor, a length adjuster, and a decoder. The phoneme embedding layer is used to obtain the phoneme vectors corresponding to the sample phoneme sequence. The encoder is used to encode the phoneme vectors to obtain intermediate vectors. The length predictor is used to predict the length of the Mel spectrum corresponding to each phoneme based on the intermediate vectors, normalize the length of the Mel spectrum corresponding to each phoneme to obtain the length ratio of each phoneme, and determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme. The length ratio is the ratio of the length of the Mel spectrum corresponding to a phoneme to the length of the standard Mel spectrum. The length adjuster is used to determine the intermediate sequence based on the intermediate vectors and the lengths of the Mel spectra corresponding to each phoneme. The decoder is used to decode the intermediate sequence to obtain the Mel spectrum corresponding to the sample phoneme sequence. The length of the intermediate sequence is the same as the length of the standard Mel spectrum. Based on the Mel spectrum corresponding to the sample phoneme sequence and the standard Mel spectrum, the acoustic model is trained to obtain the trained target acoustic model.

2. The method according to claim 1, characterized in that, The acoustic model also includes: a standard deviation predictor; The standard deviation predictor is used to obtain the standard deviation of each phoneme based on the intermediate vector. The length adjuster is specifically a soft length adjuster based on the Gaussian probability density function corresponding to each phoneme. The Gaussian probability density function corresponding to a phoneme is determined by the standard deviation of the phoneme and the center position of the phoneme. The center position of the phoneme is determined according to the length of the Mel spectrum corresponding to each phoneme.

3. The method according to claim 1 or 2, characterized in that, The acoustic model also includes: a fundamental frequency predictor and a fundamental frequency embedding layer; The base frequency predictor is used to determine the base frequency parameter sequence based on the intermediate vector, and the base frequency embedding layer is used to obtain the base frequency parameter vector corresponding to the base frequency parameter sequence. The length adjuster is specifically used to determine an intermediate sequence based on the length of the first vector and the Mel spectrum corresponding to each phoneme, wherein the first vector is the sum of the intermediate vector and the fundamental frequency parameter vector.

4. The method according to claim 1 or 2, characterized in that, The acoustic model also includes: an energy predictor and an energy embedding layer; The energy predictor is used to determine an energy parameter sequence based on the intermediate vector, and the energy embedding layer is used to obtain the energy parameter vector corresponding to the energy parameter sequence. The length adjuster is specifically used to determine an intermediate sequence based on the second vector and the length of the Mel spectrum corresponding to each phoneme, wherein the second vector is the sum of the intermediate vector and the energy parameter vector.

5. The method according to claim 4, characterized in that, The acoustic model also includes: a fundamental frequency predictor and a fundamental frequency embedding layer; The base frequency predictor is used to determine the base frequency parameter sequence based on the intermediate vector, and the base frequency embedding layer is used to obtain the base frequency parameter vector corresponding to the base frequency parameter sequence. The length adjuster is specifically used to determine the intermediate sequence based on the length of the third vector and the Mel spectrum corresponding to each phoneme, wherein the third vector is the sum of the intermediate vector, the fundamental frequency parameter vector, and the energy parameter vector.

6. A speech synthesis method, characterized in that, The method includes: Obtain the text to be synthesized speech; The length of the target Mel spectrum corresponding to the speech text to be synthesized is determined based on the text length of the speech text to be synthesized; The speech text to be synthesized is converted into a target phoneme sequence; The target phoneme sequence and the length of the target Mel spectrum are input into the target acoustic model to obtain the target Mel spectrum, and the target acoustic model is trained by the acoustic model training method as described in any one of claims 1 to 5; The target speech corresponding to the speech text to be synthesized is generated by using a vocoder to analyze the target Mel spectrum.

7. The method according to claim 6, characterized in that, The step of determining the length of the target Mel spectrum corresponding to the speech text to be synthesized includes: Based on the text length of the speech text to be synthesized, determine the initial length of the target Mel spectrum corresponding to the speech text to be synthesized; Based on the speech rate requirements of the received synthesized speech, the initial length of the target Mel spectrum is adjusted to obtain the length of the target Mel spectrum.

8. An acoustic model training device, characterized in that, include: The acquisition module is used to acquire the sample phoneme sequence, the standard Mel spectrum corresponding to the sample phoneme sequence, and the length of the standard Mel spectrum; The input module is used to input the lengths of the sample phoneme sequence and the standard Mel spectrum into the acoustic model to obtain the Mel spectrum corresponding to the sample phoneme sequence. The acoustic model includes a phoneme embedding layer, an encoder, a length predictor, a length adjuster, and a decoder. The phoneme embedding layer is used to obtain the phoneme vectors corresponding to the sample phoneme sequence. The encoder is used to encode the phoneme vectors to obtain intermediate vectors. The length predictor is used to predict the length of the Mel spectrum corresponding to each phoneme based on the intermediate vectors, normalize the length of the Mel spectrum corresponding to each phoneme to obtain the length ratio of each phoneme, and determine the length of the Mel spectrum corresponding to each phoneme based on the length of the standard Mel spectrum and the length ratio of each phoneme. The length ratio is the ratio of the length of the Mel spectrum corresponding to a phoneme to the length of the standard Mel spectrum. The length adjuster is used to determine the intermediate sequence based on the intermediate vectors and the lengths of the Mel spectra corresponding to each phoneme. The decoder is used to decode the intermediate sequence to obtain the Mel spectrum corresponding to the sample phoneme sequence. The length of the intermediate sequence is the same as the length of the standard Mel spectrum. The training module is used to train the acoustic model based on the Mel spectrum corresponding to the sample phoneme sequence and the standard Mel spectrum, so as to obtain the trained target acoustic model.

9. A speech synthesis device, characterized in that, include: The acquisition module is used to acquire the text to be synthesized speech; The determining module is used to determine the length of the target Mel spectrum corresponding to the speech text to be synthesized based on the text length of the speech text to be synthesized; The conversion module is used to convert the speech text to be synthesized into a target phoneme sequence; An input module is used to input the target phoneme sequence and the length of the target Mel spectrum into the target acoustic model to obtain the target Mel spectrum, wherein the target acoustic model is trained by the acoustic model training method as described in any one of claims 1 to 5; The generation module is used to generate the target speech corresponding to the speech text to be synthesized by using a vocoder to generate the target Mel spectrum.

10. A computer device, characterized in that, include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the acoustic model training method of any one of claims 1 to 5, or the speech synthesis method of claim 6 or 7, when the computer program is invoked.

Images