Speech synthesis method, speech synthesis device, apparatus, and storage medium

Abstract

Embodiments of the present application provide a speech synthesis method, a speech synthesis device, equipment and a storage medium, and relate to the technical field of artificial intelligence and digital medical treatment. The method comprises: obtaining a target text, then selecting a tail rhyme rhythm vector in a preset tail rhyme template library according to the sentence pattern of the target text, obtaining a feature coding vector of the target text by using a feature encoder, performing feature decoding on the feature coding vector and the tail rhyme rhythm vector by using a feature decoder to obtain a target spectrogram, and finally generating target speech of the target text according to the target spectrogram. In the speech synthesis process, the corresponding tail rhyme rhythm vector is selected according to the sentence pattern of the target text, and in the process of forming the spectrogram by feature decoding, the target speech of the target text is synthesized in combination with the tail rhyme rhythm, the naturalness of the synthesized speech is improved, and the application range of the text-to-speech synthesis technology is expanded.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and digital medical technology, and in particular to speech synthesis methods, speech synthesis devices, equipment and storage media. Background Technology

[0002] In recent years, speech synthesis technology, which converts text into speech, has been gradually applied to speech signal processing systems such as voice interaction, voice broadcasting, and personalized voice production. Simultaneously, with the rise of speech synthesis technology, it can also support the needs of digital healthcare scenarios such as health management and electronic medical records.

[0003] However, in related technologies, the speech content generated by Text-to-Speech (TTS) has a low degree of naturalness. For example, many audiobook platforms currently offer audiobook readings with synthesized speech that sounds mechanical and lacks emotion, making it difficult to achieve the intonation and rhythm of a human voice, resulting in a poor user experience. Therefore, how to improve the naturalness of speech synthesis has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to propose a speech synthesis method, speech synthesis device, equipment, and storage medium to improve the naturalness of speech synthesis.

[0005] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, comprising:

[0006] Obtain the target text and target speech identifier;

[0007] Select a prosodic vector of the final sound from a preset final sound template library based on the sentence structure of the target text and the target speech identifier;

[0008] The target text and the prosody vector of the final sound are input into a trained speech synthesis model, which includes a feature encoder, a feature decoder and a speech synthesizer.

[0009] The target text is feature-encoded using the feature encoder to obtain a feature-encoded vector.

[0010] The feature decoder is used to perform feature decoding on the feature encoding vector and the tail prosody vector to obtain the target spectrogram;

[0011] The speech synthesizer generates the target speech of the target text based on the target spectrogram.

[0012] In some embodiments, before selecting the prosodic vector of the final sound from a preset final sound template library based on the sentence structure of the target text and the target speech identifier, the following steps are included:

[0013] Construct a preset tail sound template library, the preset tail sound template library includes at least one target speech set, the target speech set corresponds to the target speech identifier, and each target speech set includes multiple basic speech signals;

[0014] Calculate the fundamental frequency information of the basic speech signal in each of the target speech sets;

[0015] The prosodic vector of the final sound of each target speech set is obtained based on the fundamental frequency information.

[0016] In some embodiments, calculating the fundamental frequency information of the underlying speech signal in each of the target speech sets includes:

[0017] The basic speech signal is processed by framing to obtain a frame speech sequence for each basic speech signal, and the frame speech sequence includes multiple speech frames.

[0018] Autocorrelation processing is performed on each of the speech frames to obtain the autocorrelation results of the speech frames;

[0019] The candidate fundamental frequency position of the speech frame is obtained based on the autocorrelation result;

[0020] Obtain the fundamental frequency information of the candidate fundamental frequency position.

[0021] In some embodiments, the final syllable prosodic vector includes one or more of a first final syllable prosodic vector, a second final syllable prosodic vector, or a third final syllable prosodic vector; obtaining the final syllable prosodic vector for each target speech set based on the fundamental frequency information includes:

[0022] Clustering the fundamental frequency information of the basic speech signal to obtain a tail sound clustering dataset, the tail sound clustering dataset including one or more of the following: a first sentence pattern dataset, a second sentence pattern dataset, or a third tail sound prosodic vector;

[0023] The first sentence pattern dataset is averaged to obtain the first final syllable prosody vector of the target speech set;

[0024] The second sentence pattern dataset is averaged to obtain the second final syllable prosody vector of the target speech set;

[0025] The third sentence pattern dataset is averaged to obtain the third final syllable prosodic vector of the target speech set.

[0026] In some embodiments, the step of using the feature encoder to perform feature encoding on the target text to obtain a feature encoding vector includes:

[0027] Obtain the corpus of target speech identifiers;

[0028] The phoneme sequence of the target text is obtained based on the corpus;

[0029] The feature encoder is used to embed features into the phoneme sequence to obtain the feature encoding vector of the phoneme sequence.

[0030] In some embodiments, the step of using the feature decoder to perform feature decoding on the feature encoding vector and the end-note prosody vector to obtain the target spectrogram includes:

[0031] The feature encoding vector and the tail phonology vector are concatenated to obtain the target synthesis vector;

[0032] The target synthesis vector is decoded to obtain the target spectrogram corresponding to the target speech identifier.

[0033] In some embodiments, the step of inputting the target text and the prosodic vector into the trained speech synthesis model further includes: training the speech synthesis model;

[0034] The training process of the speech synthesis model includes:

[0035] Obtain training samples, which include: text information and reference speech of the target speech identifier;

[0036] Select the final sound prosody vector from the preset final sound template library based on the sentence structure of the text information;

[0037] The text information is feature-encoded using the feature encoder to obtain a training encoding vector;

[0038] The feature decoder is used to perform feature decoding on the training encoded vector and the tail prosody vector to obtain a predicted spectrogram;

[0039] Calculate the prediction loss value between the predicted spectrogram and the reference spectrogram of the reference speech;

[0040] The model weights of the speech synthesis model are adjusted based on the predicted loss value to train the speech synthesis model.

[0041] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, comprising:

[0042] The acquisition unit is used to acquire the target text and the target speech identifier;

[0043] The final sound prosody selection unit is used to select a final sound prosody vector from a preset final sound template library based on the sentence structure of the target text and the target speech identifier;

[0044] An input unit is used to input the target text and the prosody vector of the final sound into a trained speech synthesis model, the speech synthesis model including a feature encoder, a feature decoder and a speech synthesizer;

[0045] A feature encoding unit is used to encode the target text using the feature encoder to obtain a feature encoding vector.

[0046] A feature decoding unit is used to perform feature decoding on the feature encoding vector and the tail phonology vector using the feature decoder to obtain a target spectrogram;

[0047] A speech generation unit is used to generate target speech of the target text based on the target spectrogram using the speech synthesizer.

[0048] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0049] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0050] The speech synthesis method, apparatus, device, and storage medium proposed in this application acquire target text, then selects a prosodic vector from a preset prosodic template library based on the sentence structure of the target text, obtains the feature encoding vector of the target text using a feature encoder, and performs feature decoding on the feature encoding vector and the prosodic vector to obtain a target spectrogram. Finally, the target speech of the target text is generated based on the target spectrogram. In this application, during speech synthesis, the corresponding prosodic vector is selected based on the sentence structure of the target text. During the process of feature decoding to form the spectrogram, the target speech of the target text is synthesized by combining the prosodic vector with the target text, thereby improving the naturalness of the synthesized speech and expanding the application scope of text-to-speech synthesis technology. Attached Figure Description

[0051] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of the present invention.

[0052] Figure 2 This is a flowchart of a speech synthesis method provided in another embodiment of the present invention.

[0053] Figure 3 yes Figure 2 The flowchart of step S210.

[0054] Figure 4 yes Figure 2 The flowchart for step S220.

[0055] Figure 5 This is a schematic diagram of the prosody of the speech synthesis method provided in the embodiment of the present invention.

[0056] Figure 6 This is a schematic diagram of the speech synthesis model of the speech synthesis method provided in the embodiment of the present invention.

[0057] Figure 7 yes Figure 1 The flowchart for step S140.

[0058] Figure 8 yes Figure 1 The flowchart for step S150.

[0059] Figure 9 This is a schematic diagram of the training process of the speech synthesis model provided in an embodiment of the present invention.

[0060] Figure 10 This is a structural block diagram of a speech synthesis device provided in another embodiment of the present invention.

[0061] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0063] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0065] First, let's clarify some of the terms used in this invention:

[0066] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0067] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that incorporate convolutional computations and have a deep structure. They are one of the representative algorithms in deep learning. CNNs possess representation learning capabilities, enabling translation-invariant classification of input information based on their hierarchical structure. CNNs are constructed by mimicking biological visual perception mechanisms and can perform both supervised and unsupervised learning. The shared parameters of the convolutional kernels within their hidden layers and the sparsity of inter-layer connections allow CNNs to process gridded features with relatively low computational cost.

[0068] The LPCNet model is based on WaveRNN and incorporates an LPC filter module to reduce neural network complexity. This model decomposes the prediction of sampling points into linear and nonlinear parts. It uses a DSP to predict the linear part and a neural network to predict the smaller nonlinear residual part.

[0069] Mel spectrogram: This is the spectrum obtained by performing a Fourier transform on an acoustic signal, followed by a Mel-scale transformation. Spectrograms are often very large images; to obtain appropriately sized sound features, they can be transformed into Mel spectrograms after passing through a Mel-scale filter bank. In the Mel frequency domain, there is a linear relationship between the Mel frequencies of speech and human pitch perception. The Mel spectrum is obtained by combining the Mel frequency cepstrum and the spectrogram.

[0070] Embedding is a commonly used feature method in the field of deep learning. It maps high-dimensional raw data (such as images, sentences, speech, etc.) to low-dimensional manifolds, making the high-dimensional raw data separable after being mapped to the low-dimensional manifold. This mapping process is called embedding.

[0071] Long Short-Term Memory (LSTM) artificial neural networks are a type of Recurrent Neural Network (RNN). LSTMs are well-suited for modeling time-series data, such as text data. The computation process of LSTM can be summarized as follows: by forgetting information in the cell state and remembering new information, useful information for subsequent time steps is passed on, while useless information is discarded. At each time step, a hidden state is output, calculated from the previous hidden state and the current input. The hidden state is controlled by forget gates, remember gates, and output gates.

[0072] In recent years, speech synthesis technology, which converts text into speech, has been increasingly applied to speech signal processing systems such as voice interaction, voice broadcasting, and personalized voice production. Effective use of speech synthesis technology can improve the user experience of voice interaction and has broad potential applications. Speech synthesis systems are widely used in various scenarios in daily life, including voice dialogue systems and intelligent voice assistants. Applications of intelligent voice assistants include telephone information query systems, in-vehicle navigation, and audio e-books; applications of voice dialogue systems include language learning systems, real-time information broadcasting systems in airports and train stations, and information acquisition and communication systems for visually or verbally impaired individuals.

[0073] The applicant found that the speech content generated by the text-to-speech (TTS) process in related technologies has low naturalness. For example, the synthesized speech content in many audiobook platforms currently available sounds mechanical and lacks emotion, making it difficult to achieve the intonation and rhythm of a human voice, resulting in a poor user experience. Therefore, improving the naturalness of speech synthesis has become an urgent technical problem to be solved.

[0074] Based on this, embodiments of the present invention provide a speech synthesis method, speech synthesis apparatus, device, and storage medium. When performing speech synthesis, the corresponding end-prosody vector is selected according to the sentence structure of the target text. During the process of feature decoding to form a spectrogram, the target speech of the target text is synthesized by combining the end-prosody, thereby improving the naturalness of the synthesized speech and expanding the application scope of text-to-speech synthesis technology.

[0075] The present invention provides a speech synthesis method, a speech synthesis apparatus, a device, and a storage medium, which are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of the present invention is described.

[0076] This invention can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0077] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0078] The speech synthesis method provided in this invention relates to the field of artificial intelligence technology, and particularly to the field of data mining technology. The speech synthesis method provided in this invention can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client that supports speech synthesis; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded into any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The speech synthesis method can be executed by the terminal or the server, or by the terminal and the server working together.

[0079] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal can also be a smart in-vehicle device. This smart in-vehicle device uses the speech synthesis method of this embodiment to provide related services and enhance the driving experience. The server can be an independent server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The server can install the server-side software of the speech synthesis system, which can interact with the terminal. For example, the server can install corresponding software, which can be an application implementing the speech synthesis method, but is not limited to the above forms. The terminal and the server can be connected via Bluetooth, USB (Universal Serial Bus), or network communication methods, and this embodiment does not impose any restrictions.

[0080] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] To facilitate understanding of the embodiments of this application, the concept of speech synthesis will be briefly introduced below with examples of specific application scenarios.

[0082] In one application scenario: A first user sends a text message and can select a target voice identifier (mainly used to identify different timbres, such as a celebrity's voice or other mechanical voices) as needed. A second user receives the corresponding audio message. When the audio message is played, the user actually hears the selected timbre voice, which also contains the prosodic information of the text message. This process utilizes text-to-speech synthesis to convert the first user's text message into corresponding audio information.

[0083] Input: The text message sent by the first user;

[0084] Output: Target speech containing prosodic information.

[0085] In another application scenario: the first user needs to listen to the audio of the novel they are reading, i.e., to achieve audio reading. The first user selects audio reading, the speech synthesis system obtains the corresponding novel text information, judges the rhythm information based on the novel text content, and incorporates the corresponding ending rhythm information during speech synthesis to generate a reading voice with intonation.

[0086] Input: The text content selected by the first user;

[0087] Output: Target speech containing prosodic information.

[0088] The speech synthesis method in the embodiments of the present invention will be described first below.

[0089] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of the present invention. Figure 1 The method may include, but is not limited to, steps S110 to S160. It is also understood that this embodiment... Figure 1 The order of steps S110 to S160 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0090] Step S110: Obtain the target text and target speech identifier.

[0091] In one embodiment, the target text information can be obtained from user input, such as chat messages, or it can be obtained by the user selecting it on the page, such as selecting the current page to be read aloud, in which case the text content of the current page is the target text information. If speech synthesis of a paragraph is required, it can be achieved by synthesizing the speech of each text in the paragraph and then summarizing them. This embodiment does not impose specific limitations on this.

[0092] In one embodiment, the target speech identifier, i.e., the timbre of the synthesized speech, can also be acquired simultaneously. For example, the timbre of a famous person can be selected as the target speech identifier. It is understood that the target speech identifier can include multiple types, and it is necessary to pre-build corpora corresponding to different target speech identifiers. The corpora contain the pronunciation of different phonemes by different target speakers.

[0093] Step S120: Select the prosodic vector of the target text from the preset end-sound template library based on the sentence structure and target speech identifier of the target text.

[0094] In one embodiment, the sentence structure of the target text includes: declarative sentences, interrogative sentences, exclamatory sentences, or other sentence structures. Different sentence structures can be determined based on the punctuation marks in the target text. For example, a period "." indicates that the target text is a declarative sentence; an interrogative mark "?" indicates that the target text is an interrogative sentence; and an exclamation mark "!" indicates that the target text is an exclamatory sentence. Alternatively, a pre-trained neural network model, such as a text semantic recognition model, can be used to identify the sentence structure of the target text based on semantics. This embodiment does not limit the method for identifying the sentence structure of the target text.

[0095] In one embodiment, a pre-built library of ending sound templates needs to be constructed. This library contains multiple target speech sets for different speakers, meaning each target speech set corresponds to a target speech identifier. Each target speech set contains multiple basic speech signals for that speaker. It is understood that the basic speech signals can be categorized according to sentence structure. For example, if there are two speakers and the sentence structures are as described above, their corresponding target speech identifiers are speaker A and speaker B. Speaker A's target speech set contains various basic speech signals for different sentence structures. Speaker A's target speech set includes: the basic speech signal for speaker A's first declarative sentence, the basic speech signal for speaker A's second declarative sentence, ..., the basic speech signal for speaker A's first interrogative sentence, the basic speech signal for speaker A's second interrogative sentence, ..., the basic speech signal for speaker A's first exclamatory sentence, the basic speech signal for speaker A's second exclamatory sentence, ... and so on. Similarly, speaker B's target speech set contains various basic speech signals for speaker B's different sentence structures, referring to the target speech set for speaker A described above.

[0096] Then, the following steps are used to calculate the prosodic vector of the final sound for each target speech set.

[0097] In one embodiment, reference is made to Figure 2 This is an embodiment illustrating the steps for calculating the prosodic vector of each target speech set, including:

[0098] Step S210: Calculate the fundamental frequency information of the basic speech signal in each target speech set.

[0099] In one embodiment, step S210 calculates the fundamental frequency information of each basic speech signal in each target speech set. The fundamental frequency is the reciprocal of the pitch period, which is the duration of each opening and closing of the vocal cords. The fundamental frequency is an important acoustic feature in speech synthesis. It is the lowest frequency pure tone in speech, but has the largest amplitude. The frequency of this pure tone is called the fundamental frequency, which determines the pitch of the entire sound.

[0100] In one embodiment, reference is made to Figure 3 This is a flowchart illustrating a specific implementation of step S210 in one embodiment. In this embodiment, step S210, which calculates the fundamental frequency information of the basic speech signal in each target speech set, may include, but is not limited to, steps S211 to S214:

[0101] Step S211: Perform frame segmentation on the basic speech signal to obtain the frame speech sequence of each basic speech signal.

[0102] In one embodiment, each frame speech sequence includes multiple speech frames.

[0103] Since the basic speech signal is an analog signal, it must be discretized before digital processing. In one embodiment, downsampling the basic speech signal using a certain sampling frequency can discretize the basic speech signal to obtain a basic discrete speech signal.

[0104] In one embodiment, the basic discrete speech signal is a non-stationary, time-varying signal. However, considering the inertial motion of the vocal organs, it can be assumed that the basic discrete speech signal is approximately invariant over a short period of time (e.g., 10ms to 30ms), meaning that the basic discrete speech signal has short-term stationary characteristics. Therefore, in one embodiment, the basic discrete speech signal is processed by framing to obtain multiple short speech frames. These speech frames can be considered as time-invariant signals over a short period of time, allowing for better quantization processing.

[0105] In one embodiment, framing is essentially windowing, which involves using a fixed-length sliding window to segment the speech signal. This window moves horizontally along the time axis, continuously segmenting the basic discrete speech signal. The length of each segmented speech frame after framing is the frame length. In one embodiment, the frame length is typically 10ms to 30ms. It is understood that if the frame length is not set appropriately during framing of the basic discrete speech signal, some speech frames may lose continuity. To avoid this, overlapping frames are inserted between two adjacent speech frames to ensure a smooth transition between them. In this embodiment, an overlapping region is set between the preceding and following speech frames. The length of this overlapping region is the frame shift, and the ratio of the frame shift to the frame length can be (0 to 1) / 2. This embodiment achieves a smooth transition between adjacent speech frames and maintains their continuity by setting a frame shift.

[0106] Step S212: Perform autocorrelation processing on each speech frame to obtain the autocorrelation result of the speech frame.

[0107] In one embodiment, the speech frame is a discrete signal. Autocorrelation is calculated for each speech frame to obtain its autocorrelation result. Since sound originates from the periodic vibration of the vocal cords, the fundamental frequency of the speech frame is periodic, and this periodicity can be used to extract the fundamental frequency. Based on this principle, an autocorrelation function is performed on the speech frame, and the corresponding frequency value is obtained by determining the maximum value of the autocorrelation function.

[0108] Step S213: Obtain the candidate fundamental frequency position of the speech frame based on the autocorrelation result.

[0109] In one embodiment, the candidate fundamental frequency position of a speech frame can be determined based on the peak point of the function in the autocorrelation result.

[0110] Step S214: Obtain the fundamental frequency information of the candidate fundamental frequency position.

[0111] In one embodiment, since there are multiple speech frames, there are also multiple candidate fundamental frequency positions, and correspondingly, multiple fundamental frequency information. Mapping the candidate fundamental frequency positions back to the basic speech signal yields multiple fundamental frequency information entries in the basic speech signal.

[0112] In one embodiment, to reduce subsequent computation, the fundamental frequency information for a preset time period is extracted from the basic speech signal. For example, the fundamental frequency information for the last 0.5 seconds of each basic speech signal is extracted. It is understood that the preset time period can be selected according to actual needs.

[0113] In one embodiment, the fundamental frequency information of the basic speech signal in each target speech set is obtained in the manner described above. That is, each target speech set contains fundamental frequency information of reference speech with different sentence structures, and each reference speech contains multiple fundamental frequency information. Since different speakers have different intonations, in order to reduce the influence of speaker information, the fundamental frequency information in each target speech set is normalized to improve the accuracy of the fundamental frequency information.

[0114] After obtaining the fundamental frequency information of the basic speech signal in each target speech set through the above steps, it is necessary to obtain the prosodic information of the final sound based on the fundamental frequency information.

[0115] Step S220: Obtain the prosodic vector of the final sound of each target speech set based on the fundamental frequency information.

[0116] In one embodiment, the types of final sound prosodic vectors correspond to the types of sentence structures. For example, if there are three sentence structures, then each target speech set contains three types of final sound prosodic vectors: a first final sound prosodic vector, a second final sound prosodic vector, or a third final sound prosodic vector, corresponding to the three sentence structures mentioned above. That is, each speaker has one final sound prosodic vector when describing the three sentence structures. It is understood that if the sentence structure only includes declarative and interrogative sentences, then the final sound prosodic vectors also contain two types.

[0117] In one embodiment, reference is made to Figure 4 This is a flowchart illustrating a specific implementation of step S220 in one embodiment. In this embodiment, step S220, which obtains the prosodic vector of the final sound of each target speech set based on the fundamental frequency information, may include, but is not limited to, steps S221 to S224:

[0118] Step S221: Cluster the fundamental frequency information of the basic speech signal to obtain the tail sound clustering dataset.

[0119] In one embodiment, the fundamental frequency information of the basic speech signals in each target speech set is clustered according to the sentence type to obtain the tail sound clustering dataset, wherein the tail sound clustering dataset includes one or more of the following: a first sentence type dataset, a second sentence type dataset, or a third tail sound prosodic vector.

[0120] For example, in one embodiment, the target speech set of speaker A includes: the basic speech signal of speaker A's first declarative sentence, the basic speech signal of speaker A's second declarative sentence, ..., the basic speech signal of speaker A's first interrogative sentence, the basic speech signal of speaker A's second interrogative sentence, ..., the basic speech signal of speaker A's first exclamatory sentence, the basic speech signal of speaker A's second exclamatory sentence, ... and so on, etc.

[0121] The above contains three sentence structures, so the clustering can produce the following datasets for the final sound clustering:

[0122] The fundamental frequency clustering result of declarative sentences is the fundamental frequency information of all declarative sentences of speaker A in the target speech set. The fundamental frequency clustering result of declarative sentences can be the first sentence pattern dataset mentioned above.

[0123] The fundamental frequency clustering result of interrogative sentences is the fundamental frequency information of all interrogative sentences of speaker A in the target speech set. The fundamental frequency clustering result of interrogative sentences can be the second sentence pattern dataset mentioned above.

[0124] The fundamental frequency clustering result of exclamatory sentences is the fundamental frequency information of all exclamatory sentences of speaker A in the target speech set. The fundamental frequency clustering result of exclamatory sentences can be the third sentence pattern dataset mentioned above.

[0125] As shown above, different sentence structures can be obtained from the clustering results. It is understandable that clustering can also be performed separately for each sentence structure within each target speech set to obtain the corresponding clustering results.

[0126] In one embodiment, since the fundamental frequency obtained in the above steps is numerical information, and normalized fundamental frequency yields normalized numerical information, clustering is a clustering of numerical vectors. Because the fundamental frequency reflects the speaker's pitch or intonation, the clustering results of different sentence patterns in the target speech set represent the pitch set of the different sentence patterns spoken by the speaker.

[0127] In one embodiment, the above clustering employs the K-means clustering algorithm, a clustering algorithm based on Euclidean distance, which considers the similarity between two values. The K-means clustering algorithm first determines the clustering parameter k, and then divides the fundamental frequency values ​​to be clustered into k clusters, ensuring that the obtained clusters satisfy the following condition: the values ​​within the same cluster are closer, and the values ​​between different clusters are more different. In this embodiment, the value of k is related to the sentence type. If the sentence type includes declarative, interrogative, and exclamatory sentences, k is 3; if the sentence type includes both declarative and interrogative sentences, k is 2. The sentence type can be selected according to actual needs. It is understood that the fundamental frequency values ​​can be represented in vector form.

[0128] Step S222: Calculate the average of the first sentence pattern dataset to obtain the first prosodic vector of the target speech set.

[0129] Step S223: Calculate the average of the second sentence pattern dataset to obtain the second end-prosody vector of the target speech set.

[0130] Step S224: Calculate the average of the third sentence pattern dataset to obtain the third end-prosody vector of the target speech set.

[0131] In one embodiment, to simplify the calculation, the values ​​or vectors obtained from the different clusters are averaged to obtain the cluster center of each cluster. The value of the cluster center is denoted as a vector and used as the prosodic vector of the corresponding dataset. In one embodiment, the first sentence dataset is averaged to obtain the first prosodic vector of the target speech set, corresponding to interrogative sentences; the second sentence dataset is averaged to obtain the second prosodic vector of the target speech set, corresponding to declarative sentences; and the third sentence dataset is averaged to obtain the third prosodic vector of the target speech set, corresponding to exclamatory sentences.

[0132] In one embodiment, since the fundamental frequency values ​​of interrogative and exclamatory sentences are greater than those of declarative sentences, the second final tone prosodic vector is denoted as a falling tone prosodic vector. The first and third final tone prosodic vectors are both denoted as rising tone prosodic vectors. However, because different speakers have different pitches for their interrogative and exclamatory sentences, some speakers have a greater fundamental frequency value for their interrogative sentences than for their exclamatory sentences, and vice versa. Therefore, the values ​​of the first and third final tone prosodic vectors differ.

[0133] Reference Figure 5 Taking the example that the fundamental frequency value of the speaker's interrogative sentence is greater than that of the fundamental frequency value of the exclamatory sentence, the prosody of the final sound is illustrated.

[0134] As shown in the graph, the horizontal axis represents the time series, and the vertical axis represents the fundamental frequency value. Since the fundamental frequency of the speaker's interrogative sentence is greater than that of the exclamatory sentence, the fundamental frequency of the prosodic vector corresponding to the first final sound of the interrogative sentence is higher than that of the prosodic vector corresponding to the third final sound of the exclamatory sentence. Because both the first and third final sound prosodic vectors belong to rising tone prosodices, their fundamental frequencies show an upward trend at the end of the speech, meaning the final sound of the sentence appears to the listener as rising. The prosodic vector corresponding to the second final sound of the declarative sentence belongs to falling tone prosodices, therefore its fundamental frequency shows a downward trend at the end of the speech, meaning the final sound of the sentence appears to the listener as falling.

[0135] Understandably, even in declarative sentences, the second final sound of different speakers follows a falling intonation rhythm; however, the fundamental frequency differs among speakers, which is reflected in... Figure 5 The difference lies in the pitch. Therefore, by constructing a preset end-sound template library using the first, second, and third end-sound prosodic vectors of each speaker, and identifying the corresponding speaker through the target speech identifier, the corresponding first, second, or third end-sound prosodic vector can be obtained from the preset end-sound template library.

[0136] For example, consider three speakers: speaker A (target speech identifier: id_1) and speaker B (target speech identifier: id_2). Speaker A's interrogative sentences have a higher fundamental frequency than their exclamatory sentences, while speaker B's interrogative sentences have a lower fundamental frequency than their exclamatory sentences. Speaker A's first, second, and third final consonant prosodic vectors are 0.81, 0.65, and 0.23, respectively; speaker B's first, second, and third final consonant prosodic vectors are 0.78, 0.91, and 0.31, respectively. The resulting preset final consonant template library is represented as follows:

[0137] {[id_1,0.81,0.65,0.23],[id_2,0.78,0.91,0.31]}

[0138] It is understood that the above-mentioned preset tail note template library is for illustrative purposes only and does not represent a limitation on it.

[0139] As can be seen from the above, after obtaining the target text and the target speech identifier, the prosodic vector of the target speech identifier can be obtained from the preset end-sound template library according to the sentence structure of the target text.

[0140] Step S130: Input the target text and the final sound prosody into the trained speech synthesis model.

[0141] In one embodiment, reference is made to Figure 6 The speech synthesis model 100 includes a feature encoder 110, a feature decoder 120, and a speech synthesizer 130. The feature encoder 110 receives the target text and is connected to a corpus 140. It is used to encode the target text based on the phoneme information of the target speech identifier in the corpus 140. Then, the feature encoding result is input into the feature decoder 120. At the same time, the feature decoder 120 receives the end-prosody vector output by the preset end-prosody template library 150 based on the sentence structure of the target text and the target speech identifier. It performs feature decoding on the feature encoding vector and the end-prosody vector and outputs the decoding result to the speech synthesizer 130 to obtain the synthesized target speech.

[0142] In one embodiment, the feature encoder 110 is a multi-head attention Transformer encoder. Correspondingly, the feature decoder 120 is also a multi-head attention Transformer encoder-decoder structure, and the speech synthesizer 130 is a vocoder implemented using the LPCNet model. This embodiment introduces an attention mechanism during speech synthesis to facilitate the joint learning of the alignment between output sound features and input sequence mappings. Adding attention within the sequence helps to find connections within the sequence, aiming to learn the dependencies within the sequence and capture its internal structure.

[0143] In one embodiment, the text information is first converted into a hidden layer representation with embedded language features by the feature encoder 110, and then transmitted to the feature decoder 120 through an attention mechanism in each decoding step. The recursive network of the feature decoder 120 accepts the hidden layer representation to predict the Mel spectrogram. Finally, the predicted Mel spectrogram is used to synthesize the target speech through the LPCNet model of the speech synthesizer 130.

[0144] Step S140: Use a feature encoder to encode the target text to obtain a feature encoding vector.

[0145] In one embodiment, reference is made to Figure 7 This is a flowchart illustrating a specific implementation of step S140 in one embodiment. In this embodiment, step S140, which uses a feature encoder to encode the target text to obtain a feature encoding vector, includes:

[0146] Step S141: Obtain the corpus of target speech identifiers.

[0147] In one embodiment, each target speech identifier includes a corresponding corpus containing the speaker's pronunciation of different phonemes.

[0148] In one embodiment, the initials and finals in Chinese are defined as 66 phonemes to form a corpus. The phonemes in the corpus are represented as follows:

[0149] "a", "aa", "ai", "an", "ang", "ao", "b", "c", "ch", "d", "e", "ee", "ei", "en", "eng", "er", "f", "g", "h", "i", "ia", "ian", "iang", "iao", "ie", "ii", "in", "ing", "iong", "iu", "ix", "iy", "iz", "j", "k", "l", "m", "n", "o", "ong", "oo", "ou", "p", "q", "r", "s", "sh", "t", "u", "ua", "uai", "uan", "uang", "ueng", "ui", "un", "uo", "uu", "v", "van", "ve", "vn", "vv", "x", "z", "zh".

[0150] Step S142: Obtain the phoneme sequence of the target text based on the corpus.

[0151] In one embodiment, first convert the target text into word segmentation information, then use the obtained target voice identifier to select the corresponding corpus, and obtain the phoneme composition of each word segmentation in the corpus as the phoneme sequence. For example, the voice object identifier selected by the user is: id_1, and the input target text is "The weather is very good today". First, convert it into word segmentation information: "jintian tian qi hen hao", and then convert the word segmentation information into a phoneme sequence, expressed as: {j, in, t, i, an, t, i, an, q, i, h, en, h, ao}, where the pronunciation of each phoneme in the phoneme sequence is the pronunciation information of the speaker corresponding to this id_1.

[0152] Step S143: Use the feature encoder to perform feature embedding on the phoneme sequence to obtain the feature encoding vector of the phoneme sequence.

[0153] Embedding is a feature processing method that maps high-dimensional raw data (such as images, sentences, voices, etc.) to a low-dimensional manifold, making the high-dimensional raw data separable after being mapped to the low-dimensional manifold. In one embodiment, step S143 uses the feature encoder to perform an embedding operation on the input phoneme sequence to obtain the corresponding feature encoding vector, and the feature encoding vector is a low-dimensional feature vector. The purpose of the feature encoder is to convert the phoneme sequence into a data format that the computer can use for calculation.

[0154] Step S150: Use the feature decoder to perform feature decoding on the feature encoding vector and the coda prosody vector to obtain the target spectrogram.

[0155] Step S160: The speech synthesizer generates the target voice of the target text according to the target spectrogram.

[0156] In one embodiment, the feature decoder performs feature decoding on the feature encoding vector and the tail prosody vector to obtain a target spectrogram, which is a Mel spectrum.

[0157] In one embodiment, reference is made to Figure 8 This is a flowchart illustrating a specific implementation of step S150 in one embodiment. In this embodiment, step S150, which uses a feature decoder to perform feature decoding on the feature encoding vector and the tail prosody vector to obtain the target spectrogram, includes:

[0158] Step S151: Concatenate the feature encoding vector and the final syllable prosody vector to obtain the target synthesized vector.

[0159] Step S152: Decode the target synthesis vector to obtain the target spectrogram corresponding to the target speech identifier.

[0160] In one embodiment, the feature encoding vector and the prosodic vector are concatenated, and then the concatenated target synthesis vector is upsampled. The corresponding spectrogram is then selected and concatenated according to the sampling result to complete the decoding process and obtain the target spectrogram of the phoneme sequence corresponding to the target speech identifier.

[0161] In one embodiment, step S160 uses a speech synthesizer to convert the Mel spectrum into corresponding speech content. The speech content is a wave file that represents the speech signal in the form of a wave. This embodiment does not limit the form of the speech content.

[0162] As described above, the speech synthesis method proposed in this application involves acquiring target text, selecting end-note prosody vectors from a preset end-note template library based on the sentence structure of the target text, obtaining feature encoding vectors of the target text using a feature encoder, and performing feature decoding on the feature encoding vectors and end-note prosody vectors to obtain a target spectrogram. Finally, the target speech of the target text is generated based on the target spectrogram. By combining end-note prosody with the synthesis of target speech from the target text, the naturalness of the synthesized speech is improved, expanding the application scope of text-to-speech synthesis technology.

[0163] The training process of the speech synthesis model in the embodiments of this application is described below.

[0164] Reference Figure 9 The training process of the speech synthesis model in this application embodiment includes the following steps:

[0165] Step S910: Obtain training samples.

[0166] In one embodiment, the training samples include: text information and reference speech of the target speech identifier. It is understood that the reference speech here includes end-tone prosody information. For example, the text information of training sample 1 is: “Have you eaten?”, and the target speech identifier is: speaker A (id_1). Based on the text information, the sentence structure is interrogative, therefore the end-tone prosody is a rising tone prosody, and the reference speech is speaker A speaking “Have you eaten?” in a rising tone.

[0167] Step S920: Select the ending sound prosody vector from the preset ending sound template library according to the sentence structure of the text information.

[0168] In one embodiment, the preset tail sound template library needs to be pre-trained through clustering. It is understood that speaker information can be updated or added to the preset tail sound template library as needed to expand the application scenarios of speech synthesis.

[0169] Step S930: Use a feature encoder to encode the text information to obtain a training encoding vector.

[0170] Step S940: Use the feature decoder to perform feature decoding on the training encoding vector and the end-prosody vector to obtain the predicted spectrogram.

[0171] The feature encoding and feature decoding processes described above are as described in the above embodiments.

[0172] Step S950: Calculate the prediction loss between the predicted spectrogram and the reference spectrogram of the reference speech.

[0173] In one embodiment, a reference spectrogram of the reference speech is first obtained, for example, a reference Mel spectrum of the reference speech, denoted as . The predicted spectrogram is denoted as

[0174] Step S960: Adjust the model weights of the speech synthesis model according to the predicted loss value to train the speech synthesis model.

[0175] In one embodiment, the prediction loss value is denoted as:

[0176]

[0177] Where n represents the number of sampling points in the Mel spectrum.

[0178] In one embodiment, the model weights of the speech synthesis model are adjusted so that the prediction loss value meets a convergence condition. This convergence condition can be reaching a preset number of iterations or the loss value being minimized within a certain range. This embodiment does not specifically limit the convergence condition for the prediction loss value. It is understood that the smaller the prediction loss value, the better the reference Mel spectrum. and predicted spectrogram The closer they are, the more likely they are to produce nearly identical audio content.

[0179] As can be seen from the above, the trained speech synthesis model only needs to input the target text and the target speech identifier to generate the correct speech content and the correct ending prosody.

[0180] The technical solution provided by this invention involves acquiring target text, selecting a tail sound prosody vector from a preset tail sound template library based on the sentence structure of the target text, obtaining the feature encoding vector of the target text using a feature encoder, performing feature decoding on the feature encoding vector and the tail sound prosody vector using a feature decoder to obtain a target spectrogram, and finally generating the target speech of the target text based on the target spectrogram.

[0181] The speech synthesis method of this application embodiment can be used in the medical field. The target text is medical data collected with the patient's informed consent, such as personal health records, prescriptions, examination reports, etc. The application scenarios of the speech synthesis method of this application embodiment in the digital medical field are described below.

[0182] In hospitals and other medical institutions, speech synthesis technology can enable robots or other intelligent terminals to converse with patients like humans, providing necessary guidance, advice, and services. The speech synthesis method described in this application achieves high naturalness in the synthesized speech of robots or other intelligent terminals. This not only alleviates the heavy workload of medical staff but also better serves patients and improves their user experience during communication. Alternatively, medical staff can use speech synthesis technology to convert medical records into speech, which can then be saved and backed up in a data center. This method not only speeds up the recording process but, more importantly, makes the documents more realistic and vivid, thus providing medical staff with more accurate information.

[0183] In this embodiment of the application, during speech synthesis, the corresponding end-pronunciation vector is selected based on the sentence structure of the target text. During feature decoding to form a spectrogram, the end-pronunciation is combined to synthesize the target speech of the target text. This end-pronunciation control method is relatively simple, reducing the complexity of the speech synthesis model, improving the synthesis efficiency, and enhancing the naturalness of the synthesized speech, thus expanding the application scope of text-to-speech synthesis technology. For example, more diverse end-pronunciation can provide audiobook users with a more nuanced reading experience.

[0184] This invention also provides a speech synthesis apparatus that can implement the above-described speech synthesis method, see reference. Figure 10 The device includes:

[0185] The acquisition unit 1010 is used to acquire the target text and the target speech identifier.

[0186] The final sound prosody selection unit 1020 is used to select a final sound prosody vector from a preset final sound template library based on the sentence structure of the target text and the target speech identifier.

[0187] The input unit 1030 is used to input the target text and the prosody vector of the final sound into the trained speech synthesis model, which includes a feature encoder, a feature decoder and a speech synthesizer.

[0188] The feature encoding unit 1040 is used to encode the target text using a feature encoder to obtain a feature encoding vector.

[0189] The feature decoding unit 1050 is used to perform feature decoding on the feature encoding vector and the end-note prosody vector to obtain the target spectrogram.

[0190] The speech generation unit 1060 is used to generate target speech of target text based on target spectrogram using a speech synthesizer.

[0191] The specific implementation of the speech synthesis device in this embodiment is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0192] This invention also provides an electronic device, comprising:

[0193] At least one memory;

[0194] At least one processor;

[0195] At least one program;

[0196] The program is stored in a memory, and the processor executes the at least one program to implement the speech synthesis method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0197] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0198] The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0199] The memory 1102 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101 to execute the speech synthesis method of the embodiments of this invention.

[0200] Input / output interface 1103 is used to implement information input and output;

[0201] Communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0202] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);

[0203] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0204] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described speech synthesis method.

[0205] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0206] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this invention acquire target text, then select a prosodic vector from a preset prosodic template library based on the sentence structure of the target text, obtain the feature encoding vector of the target text using a feature encoder, and perform feature decoding on the feature encoding vector and the prosodic vector to obtain a target spectrogram. Finally, the target speech of the target text is generated based on the target spectrogram. In this embodiment, during speech synthesis, the corresponding prosodic vector is selected based on the sentence structure of the target text. During the process of feature decoding to form the spectrogram, the target speech of the target text is synthesized by combining the prosodic vector with the target text, thereby improving the naturalness of the synthesized speech and expanding the application scope of text-to-speech synthesis technology.

[0207] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0208] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0209] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0210] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0211] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0212] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0214] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0216] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the target text and target speech identifier; Select a prosodic vector of the final sound from a preset final sound template library based on the sentence structure of the target text and the target speech identifier; The target text and the prosody vector of the final sound are input into a trained speech synthesis model, which includes a feature encoder, a feature decoder and a speech synthesizer. The target text is feature-encoded using the feature encoder to obtain a feature-encoded vector. The feature decoder is used to perform feature decoding on the feature encoding vector and the tail prosody vector to obtain the target spectrogram; The speech synthesizer is used to generate the target speech of the target text based on the target spectrogram. The method further includes, before selecting the prosodic vector of the final sound from the preset final sound template library based on the sentence structure of the target text and the target speech identifier: Construct a preset tail sound template library, the preset tail sound template library includes at least one target speech set, the target speech set corresponds to the target speech identifier, and each target speech set includes multiple basic speech signals; Calculate the fundamental frequency information of the basic speech signal in each of the target speech sets; The prosodic vector of the final sound of each target speech set is obtained based on the fundamental frequency information.

2. The speech synthesis method according to claim 1, characterized in that, The calculation of the fundamental frequency information of the basic speech signal in each of the target speech sets includes: The basic speech signal is processed by framing to obtain a frame speech sequence for each basic speech signal, and the frame speech sequence includes multiple speech frames. Autocorrelation processing is performed on each of the speech frames to obtain the autocorrelation results of the speech frames; The candidate fundamental frequency position of the speech frame is obtained based on the autocorrelation result; Obtain the fundamental frequency information of the candidate fundamental frequency position.

3. The speech synthesis method according to claim 2, characterized in that, The final syllable prosodic vector includes one or more of a first final syllable prosodic vector, a second final syllable prosodic vector, or a third final syllable prosodic vector; obtaining the final syllable prosodic vector for each target speech set based on the fundamental frequency information includes: Clustering the fundamental frequency information of the basic speech signal to obtain a tail sound clustering dataset, the tail sound clustering dataset including one or more of the following: a first sentence pattern dataset, a second sentence pattern dataset, or a third sentence pattern dataset; The first sentence pattern dataset is averaged to obtain the first final syllable prosody vector of the target speech set; The second sentence pattern dataset is averaged to obtain the second final syllable prosody vector of the target speech set; The third sentence pattern dataset is averaged to obtain the third final syllable prosodic vector of the target speech set.

4. The speech synthesis method according to claim 1, characterized in that, The step of using the feature encoder to encode the target text to obtain a feature encoding vector includes: Obtain the corpus of target speech identifiers; The phoneme sequence of the target text is obtained based on the corpus; The feature encoder is used to embed features into the phoneme sequence to obtain the feature encoding vector of the phoneme sequence.

5. The speech synthesis method according to claim 4, characterized in that, The step of using the feature decoder to perform feature decoding on the feature encoding vector and the tail prosody vector to obtain the target spectrogram includes: The feature encoding vector and the tail phonology vector are concatenated to obtain the target synthesis vector; The target synthesis vector is decoded to obtain the target spectrogram corresponding to the target speech identifier.

6. The speech synthesis method according to any one of claims 1-5, characterized in that, Before inputting the target text and the prosodic vector into the trained speech synthesis model, the method further includes: training the speech synthesis model; The training process of the speech synthesis model includes: Obtain training samples, which include: text information and reference speech of the target speech identifier; Select the final sound prosody vector from the preset final sound template library based on the sentence structure of the text information; The text information is feature-encoded using the feature encoder to obtain a training encoding vector; The feature decoder is used to perform feature decoding on the training encoded vector and the tail prosody vector to obtain a predicted spectrogram; Calculate the prediction loss value between the predicted spectrogram and the reference spectrogram of the reference speech; The model weights of the speech synthesis model are adjusted based on the predicted loss value to train the speech synthesis model.

7. A speech synthesis device, characterized in that, The apparatus is used to implement the speech synthesis method according to any one of claims 1 to 6, the apparatus comprising: The acquisition unit is used to acquire the target text and the target speech identifier; The final sound prosody selection unit is used to select a final sound prosody vector from a preset final sound template library based on the sentence structure of the target text and the target speech identifier; An input unit is used to input the target text and the prosody vector of the final sound into a trained speech synthesis model, the speech synthesis model including a feature encoder, a feature decoder and a speech synthesizer; A feature encoding unit is used to encode the target text using the feature encoder to obtain a feature encoding vector. A feature decoding unit is used to perform feature decoding on the feature encoding vector and the tail phonology vector using the feature decoder to obtain a target spectrogram; A speech generation unit is used to generate target speech of the target text based on the target spectrogram using the speech synthesizer.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 6.

Images