Artificial intelligence-based speech synthesis method and device, computer equipment and medium

Abstract

The application is suitable for the medical technology field, and particularly relates to a speech synthesis method and device based on artificial intelligence, computer equipment and medium. The speech text features of target speech text are extracted through a text encoder, the context attention features of context text are extracted through a context encoder, the prosody features of reference audio are extracted through a prosody encoder, first fusion features are obtained through fusion, and prosody information under the influence of context text is extracted; predicted prosody factor features of the first fusion features are extracted through a prosody factor predictor, second fusion features are obtained through fusion of the first fusion features and the predicted prosody factor features, target synthesized speech is obtained according to the second fusion features, the richness and accuracy of prosody information are improved by combining the duration, pitch and energy of sound, the accuracy of the target synthesized speech is improved, the medical personnel are efficiently assisted in handling repetitive work in the medical technology field, and the work efficiency and quality of the medical personnel are improved.

Description

Artificial intelligence-based speech synthesis methods, devices, computer equipment, and media Technical Field

[0001] This invention relates to the field of medical technology, and in particular to a speech synthesis method, device, computer equipment and medium based on artificial intelligence. Background Technology

[0002] Due to the complexity of medical work and the large amount of repetitive basic tasks required of medical staff, intelligent voice technology has been increasingly widely used in the medical field in recent years. With the continuous development of Internet big data and voice synthesis technology, voice synthesis technology, as an assistant in doctors' work, can help doctors handle a large amount of tedious and repetitive labor with its characteristics of speed, accuracy and low error rate, greatly reducing the burden on medical staff.

[0003] The goal of speech synthesis technology is to convert text information into speech signals. It is a highly upsampled process. There is a one-to-many mapping between text and speech data pairs. Speech data not only contains the corresponding text information, but also the corresponding prosodic information. Modeling the prosodic information is crucial for synthesizing natural and expressive speech.

[0004] Existing speech synthesis methods generally extract textual and prosodic features from the text to be synthesized, concatenate these features, and then perform feature decoding to synthesize the target speech. However, the applicant discovered that the prosodic information in speech is affected by the context of the text to be synthesized. The aforementioned speech synthesis methods only focus on the current text to be synthesized while ignoring the text content of the preceding and following sentences, resulting in low accuracy of the extracted prosodic features and reducing the accuracy of the speech synthesis results.

[0005] Therefore, in the field of medical technology, improving the accuracy of speech synthesis results has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a speech synthesis method, apparatus, computer equipment and medium based on artificial intelligence to solve the problem of low accuracy of existing speech synthesis results.

[0007] In a first aspect, embodiments of the present invention provide an artificial intelligence-based speech synthesis method, the speech synthesis method comprising:

[0008] Obtain the target speech text, the context text of the target speech text, and the reference audio; input the target speech text into a trained text encoder to obtain speech text features; input the context text into a trained context encoder to obtain context attention features; input the reference audio into a trained prosodic encoder to obtain prosodic features.

[0009] The speech-text features, the contextual attention features, and the prosodic features are fused to obtain the first fused feature;

[0010] The first fused feature is input into the trained prosodic factor predictor to obtain the predicted prosodic factor feature, wherein the prosodic factor of the predicted prosodic factor feature includes at least one of the duration of the sound, the pitch of the sound, and the energy of the sound.

[0011] The first fusion feature and the predicted prosodic factor feature are fused to obtain the second fusion feature;

[0012] The second fusion feature is input into the trained decoder to obtain the Mel spectrum, the Mel spectrum is input into the trained vocoder to obtain the sound waveform, and the target synthesized speech is obtained based on the sound waveform.

[0013] Secondly, embodiments of the present invention provide an artificial intelligence-based speech synthesis device, the speech synthesis device comprising:

[0014] The first feature extraction module is used to acquire the target speech text, the context text of the target speech text, and the reference audio. The target speech text is input into a trained text encoder to obtain speech text features. The context text is input into a trained context encoder to obtain context attention features. The reference audio is input into a trained prosodic encoder to obtain prosodic features.

[0015] The first feature fusion module is used to fuse the speech text features, the context attention features, and the prosodic features to obtain the first fused feature;

[0016] The second feature extraction module is used to input the first fused feature into the trained prosodic factor predictor to obtain predicted prosodic factor features. The prosodic factors of the predicted prosodic factor features include at least one of the duration of the sound, the pitch of the sound, and the energy of the sound.

[0017] The second feature fusion module is used to perform feature fusion on the first fused feature and the predicted prosodic factor feature to obtain the second fused feature;

[0018] The target speech synthesis module is used to input the second fused feature into a trained decoder to obtain a Mel spectrum, input the Mel spectrum into a trained vocoder to obtain a sound waveform, and obtain the target synthesized speech based on the sound waveform.

[0019] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech synthesis method as described in the first aspect.

[0020] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis method as described in the first aspect.

[0021] The beneficial effects of this invention compared to existing technologies are as follows: By inputting the target speech text into a trained text encoder to obtain speech text features, inputting the context text into a trained context encoder to obtain context attention features, and inputting the reference audio into a trained prosodic encoder to obtain prosodic features, the speech text features, context attention features, and prosodic features are fused to obtain a first fused feature, effectively extracting the text information of the text to be synthesized, as well as the prosodic information under the influence of the context text; the first fused feature is then input into a trained prosodic factor predictor to obtain predicted prosodic factor features, and the first fused feature and predicted prosodic factor features are fused to obtain a second fused feature, which, combined with prosodic factors such as the duration, pitch, and energy of the sound, further improves the richness and accuracy of the prosodic information; finally, the second fused feature is input into a trained decoder to obtain a Mel spectrum, the Mel spectrum is input into a trained vocoder to obtain a sound waveform, and the target synthesized speech is obtained based on the sound waveform, improving the accuracy of the target synthesized speech. In the field of medical technology, this invention efficiently assists medical staff in accurately handling a large amount of tedious and repetitive work, greatly improving the work efficiency and quality of medical staff. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 is a schematic diagram of an application environment for an artificial intelligence-based speech synthesis method provided in Embodiment 1 of the present invention;

[0024] Figure 2 is a flowchart illustrating an artificial intelligence-based speech synthesis method provided in Embodiment 1 of the present invention;

[0025] Figure 3 is a schematic diagram of the structure of an artificial intelligence-based speech synthesis device provided in Embodiment 2 of the present invention;

[0026] Figure 4 is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0028] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0031] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0034] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0035] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0036] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0037] The first embodiment of this invention provides an artificial intelligence-based speech synthesis method, which can be applied in the application environment shown in Figure 1, wherein the client and the server communicate with each other. The client includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other computer devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0038] Referring to Figure 2, which is a flowchart illustrating an artificial intelligence-based speech synthesis method according to Embodiment 1 of the present invention, the speech synthesis method can be applied to the client shown in Figure 1. The speech synthesis method may include the following steps:

[0039] Step S201: Obtain the target speech text, the context text of the target speech text, and the reference audio. Input the target speech text into the trained text encoder to obtain speech text features. Input the context text into the trained context encoder to obtain context attention features. Input the reference audio into the trained prosodic encoder to obtain prosodic features.

[0040] The goal of speech synthesis technology is to convert text information into corresponding speech signals. Therefore, this embodiment requires obtaining the target speech text to be synthesized. By extracting the speech text features from the target speech text, the textual information basis for the speech synthesis result is obtained. In the field of medical technology, the target speech text and contextual text can be disease-related popular science text, and the reference audio is the audio corresponding to the text content of the target speech text. By analyzing the target speech text, contextual text, and reference audio, the target synthesized speech can be obtained, efficiently assisting medical workers in popularizing disease knowledge, reducing their workload, and thus improving their work efficiency and quality.

[0041] Since speech synthesis is a highly upsampled process, the text-speech data pairs have a one-to-many mapping. The speech data contains not only the corresponding text information but also the corresponding prosodic information. Therefore, to improve the naturalness and expressiveness of the speech synthesis results, this embodiment also needs to obtain the reference audio of the target speech text. By extracting the prosodic features from the reference audio, the prosodic information basis of the speech synthesis result is obtained. The reference audio of the target speech text is the audio corresponding to the text content of the target speech text.

[0042] Since the prosodic information in the speech synthesis result is affected by the context of the target speech text, this embodiment also needs to obtain the context text of the target speech text and extract context attention features to improve the accuracy of the prosodic information of the speech synthesis result. The number of characters in the context text can be set according to the actual situation. For example, the preceding text with the same number of characters as the target speech text and the following text with the same number of characters can be selected, and the corresponding preceding and following texts can be obtained based on the corresponding preceding and following texts.

[0043] Specifically, after obtaining the target speech text, the context text of the target speech text, and the reference audio, the target speech text is input into a trained text encoder for feature extraction to obtain speech text features. The context text is input into a trained context encoder for feature extraction to obtain context attention features. The reference audio is input into a trained prosodic encoder for feature extraction to obtain prosodic features. The extracted speech text features, context attention features, and prosodic features are used as the basis for speech synthesis.

[0044] Optionally, the trained context encoder includes a trained word embedding layer, a trained bidirectional gated recurrent layer, and a trained attention mechanism layer;

[0045] The context text is input into the trained context encoder to obtain the context attention features, including:

[0046] The context text is input into the trained word embedding layer to embed word vectors, thus obtaining word embedding features;

[0047] The word embedding features are input into a trained bidirectional gated recurrent layer for feature association to obtain contextual features;

[0048] The contextual features are input into the trained attention mechanism layer for feature interaction to obtain contextual attention features.

[0049] The trained context encoder is used to extract features from the context text, obtaining context attention features that contain prosodic information within the context text. In this embodiment, the context encoder includes a word embedding layer, a bidirectional gated recurrent layer, and an attention mechanism layer.

[0050] Specifically, word embedding is a method that uses dense vectors to represent words and documents. A word embedding layer can transform the text input to the layer into a dense vector form. In this embodiment, the context text is input to the trained word embedding layer for word vector embedding, transforming the context text into a dense vector form, thus obtaining the word embedding features of the context text.

[0051] Gated recurrent layers are a commonly used type of gated recurrent neural network (RNN). They help capture long-term dependencies in time series and can solve the gradient vanishing or exploding problems found in standard RNNs, while preserving long-term information of the sequence. In this embodiment, word embedding features are input into a trained bidirectional gated recurrent layer for feature association, resulting in forward context output features and backward context output features. These forward and backward context output features are then concatenated to obtain context features.

[0052] Attention mechanisms are special structures embedded in machine learning models to interact with the input data of a layer, automatically learning and calculating the contribution of the input data to the output data. In this embodiment, contextual features are input to the trained attention mechanism layer for feature interaction. Attention scores are calculated and weighted sums are performed on these contextual features to obtain contextual attention features, thereby improving the accuracy of contextual attention feature extraction.

[0053] Optionally, the trained prosodic encoder includes a trained first convolutional layer, a trained gated recurrent layer, and a trained first fully connected layer;

[0054] The reference audio is input into the trained prosodic encoder to obtain prosodic features, including:

[0055] The reference audio is input into the trained first convolutional layer for feature extraction to obtain the first prosodic feature.

[0056] The first prosodic feature is input into the trained gated recurrent layer for feature association to obtain the second prosodic feature;

[0057] The second prosodic feature is input into the trained first fully connected layer for feature mapping to obtain the prosodic feature.

[0058] Among them, the convolutional layer is used to extract the features of the input data of the layer through convolution operation; the gated recurrent layer helps to capture long-term dependencies in the time series, which can solve the gradient vanishing or gradient explosion problem in the standard recurrent neural network, while preserving the long-term information of the sequence; the fully connected layer is used to map the input data of the layer to the hidden layer feature space, reduce the dimensionality of the output data, and reduce the parameter complexity in the model.

[0059] Specifically, in this embodiment, the reference audio is input into a trained first convolutional layer for feature extraction to obtain a first prosodic feature. Then, the first prosodic feature is input into a trained gated recurrent layer for feature association to capture long-term dependencies in the first prosodic feature and obtain a second prosodic feature. The second prosodic feature is then input into a trained first fully connected layer for feature mapping to reduce the dimensionality of the second prosodic feature and obtain the prosodic feature, thereby improving the accuracy of prosodic feature extraction.

[0060] The steps described above—obtaining the target speech text, the context text of the target speech text, and the reference audio; inputting the target speech text into a trained text encoder to obtain speech text features; inputting the context text into a trained context encoder to obtain context attention features; and inputting the reference audio into a trained prosodic encoder to obtain prosodic features—extract features from the target speech text, context text, and reference audio respectively, resulting in speech text features, context attention features, and prosodic features. These serve as the feature basis for textual and prosodic information in the speech synthesis result. By combining context attention features and prosodic features, the prosodic information is fully represented, thereby improving the accuracy of prosodic information in the speech synthesis result.

[0061] Step S202: The speech text features, contextual attention features, and prosodic features are fused to obtain the first fused feature.

[0062] Among them, the speech-text features contain textual information in the target speech-text, while the contextual attention features and prosodic features contain prosodic information in both the target speech-text and the contextual text. When the target speech data contains the corresponding textual and prosodic information, the synthesized target speech is more natural and expressive.

[0063] Feature fusion can comprehensively utilize multiple features to achieve complementary advantages of multiple features and obtain more robust and accurate recognition results. Feature fusion algorithms can be mainly divided into three categories: algorithms based on Bayesian decision theory, algorithms based on sparse representation theory, and algorithms based on deep learning theory. The specific feature fusion algorithm can be selected according to the actual situation.

[0064] In this embodiment, in order to improve the accuracy of the speech synthesis results, the speech text features, context attention features and prosodic features are fused to obtain a first fused feature that integrates text information and prosodic information, which serves as the synthesis basis for the speech synthesis results.

[0065] The above steps of fusing speech-text features, contextual attention features, and prosodic features to obtain the first fused feature, which integrates textual information and prosodic information, achieves complementarity between speech-text features, contextual attention features, and prosodic features, and serves as the basis for speech synthesis results, effectively improving the accuracy of speech synthesis results.

[0066] Step S203: Input the first fused feature into the trained prosodic factor predictor to obtain the predicted prosodic factor feature. The prosodic factor of the predicted prosodic factor feature includes at least one of the duration of the sound, the pitch of the sound, and the energy of the sound.

[0067] In addition to the timbre units of vowels and consonants arranged in chronological order, speech also includes the duration, pitch, and energy of sounds—that is, the prosodic factors of speech. These prosodic factors influence the prosodic characteristics of speech at different levels. For example, at the phoneme level, variations in sound duration can create long and short sounds that distinguish meaning; at the syllable level, variations in pitch can create tones; at the syllable combination level, variations in factors such as sound energy can create stress; and at the sentence level, variations in prosodic factors such as pitch, intensity, and duration can create intonation.

[0068] Therefore, in this embodiment, in order to improve the richness and accuracy of prosodic information in the speech synthesis result, the first fusion feature is input into the trained prosodic factor predictor for feature extraction to obtain predicted prosodic factor features, so as to predict the duration, pitch, energy and other prosodic factors of the sound in the speech synthesis result, thereby improving the accuracy of the speech synthesis result.

[0069] Optionally, the trained prosodic factor predictor includes a trained second convolutional layer, a trained normalized layer, and a trained second fully connected layer;

[0070] The first fusion feature is input into the trained prosodic factor predictor to obtain the predicted prosodic factor features, including:

[0071] The first fused feature is input into the trained second convolutional layer for feature extraction to obtain the first prosodic factor feature;

[0072] The first prosodic feature is input into the trained standardization layer for feature standardization to obtain the second prosodic feature.

[0073] The second prosodic feature is input into the trained second fully connected layer for feature mapping to obtain the predicted prosodic feature.

[0074] The convolutional layer is used to extract features from the input data of the layer through convolution operations; the normalization layer is used to transform the input data of the layer into standard data with a mean of 0 and a variance of 1. By adjusting the data distribution, the effectiveness of the gradient is ensured, which can speed up the training speed of the prosodic factor predictor and improve the accuracy of the prosodic factor predictor; the fully connected layer is used to map the input data of the layer to the hidden layer feature space, reduce the dimensionality of the output data, and reduce the parameter complexity in the model.

[0075] Specifically, in this embodiment, the first fusion feature is input into the trained second convolutional layer for feature extraction to obtain the first prosodic feature. Then, the first prosodic feature is input into the trained normalization layer for feature normalization to obtain the second prosodic feature. The second prosodic feature is input into the trained second fully connected layer for feature mapping to obtain the predicted prosodic feature. By performing feature extraction, feature normalization, and feature mapping on the first fusion feature, the predicted prosodic feature is extracted as the prosodic basis for speech synthesis, thereby improving the richness and accuracy of prosodic information in the speech synthesis result.

[0076] Optionally, the trained prosodic factor predictor includes a trained duration predictor.

[0077] The first fusion feature is input into the trained prosodic factor predictor to obtain the predicted prosodic factor features, including:

[0078] The first fusion feature is input into the trained duration predictor to obtain the predicted duration feature, which is used as the predicted prosodic factor feature.

[0079] The variation in the duration of a sound can constitute long and short sounds that have a distinguishing effect. In this embodiment, the duration of a sound, as a prosodic factor, affects the prosodic information in the speech synthesis result. Therefore, the trained prosodic factor predictor includes a trained duration predictor. The first fusion feature is input into the trained duration predictor for feature extraction to obtain the predicted duration feature, which is used as the predicted prosodic factor feature to predict the duration of the sound in the speech synthesis result, thereby improving the accuracy of the speech synthesis result.

[0080] Optionally, the trained prosodic factor predictor also includes a trained pitch predictor;

[0081] The first fused feature is input into the trained prosodic factor predictor to obtain the predicted prosodic factor features, which also include:

[0082] The first fusion feature is input into the trained pitch predictor to obtain the predicted pitch feature. The predicted duration feature and the predicted pitch feature are used together as the predicted prosodic factor feature.

[0083] The pitch of a sound can constitute a tone. In this embodiment, the pitch of a sound, as a prosodic factor, affects the prosodic information in the speech synthesis result. Therefore, the trained prosodic factor predictor also includes a trained pitch predictor. The first fusion feature is input into the trained pitch predictor to obtain the predicted pitch feature. Then, the predicted duration feature and the predicted pitch feature can be used together as the predicted prosodic factor feature to predict the duration and pitch of the sound in the speech synthesis result, thereby improving the accuracy of the speech synthesis result.

[0084] Optionally, the trained prosody factor predictor also includes a trained energy predictor;

[0085] The first fused feature is input into the trained prosodic factor predictor to obtain the predicted prosodic factor features, which also include:

[0086] The first fusion feature is input into the trained energy predictor to obtain the predicted energy feature. The predicted duration feature, predicted pitch feature, and predicted energy feature are used together as the predicted prosodic factor feature.

[0087] Among them, changes in factors such as sound energy can constitute stress and emphasis. Sound energy, as a prosodic factor, affects the prosodic information in the speech synthesis result. Therefore, the trained prosodic factor predictor also includes a trained energy predictor. The first fusion feature is input into the trained energy predictor to obtain the predicted energy feature. Then, the predicted duration feature, predicted pitch feature, and predicted energy feature can be used together as the predicted prosodic factor feature to predict the duration, pitch, and energy of the sound in the speech synthesis result, thereby improving the accuracy of the speech synthesis result.

[0088] In one embodiment, the predicted prosodic features can be composed of any one of the predicted duration features, predicted pitch features, and predicted energy features, which respectively affect the prosodicity of the speech synthesis results in terms of vowel length, tone, and stress. Alternatively, they can be composed of any two of the predicted duration features, predicted pitch features, and predicted energy features, combined with any two of the vowel length, tone, and stress aspects to affect the prosodicity of the speech synthesis results. Or, they can be composed of the predicted duration features, predicted pitch features, and predicted energy features together, combined with vowel length, tone, stress, and intonation aspects to affect the prosodicity of the speech synthesis results, thereby improving the richness and accuracy of prosodic information in the speech synthesis results and thus improving the accuracy of the speech synthesis results.

[0089] The above steps involve inputting the first fusion feature into the trained prosodic factor predictor to obtain predicted prosodic factor features. The prosodic factors of the predicted prosodic factor features include at least one of the duration of the sound, the pitch of the sound, and the energy of the sound. Based on the first fusion feature, the predicted prosodic factor features are extracted, which improves the richness and accuracy of prosodic information in the speech synthesis results, thereby improving the accuracy of the speech synthesis results.

[0090] Step S204: Perform feature fusion on the first fusion feature and the predicted prosodic factor feature to obtain the second fusion feature.

[0091] The first fusion feature is obtained by fusing speech text features, contextual attention features and prosodic features, which contains textual information of the target speech text, as well as prosodic information in the target speech text and contextual text. The predictive prosodic factor features are extracted based on the first fusion feature to predict prosodic factor features that affect prosodic information, further improving the richness and accuracy of prosodic information.

[0092] To further improve the accuracy of speech synthesis results, this embodiment fuses the first fusion feature and the predicted prosodic factor feature to obtain the second fusion feature. The second fusion feature can effectively represent the text information of the target speech text, the prosodic information in the target speech text and the context text, as well as the predicted prosodic factor information that affects the prosodic information. This second fusion feature serves as the basis for speech synthesis and improves the accuracy of speech synthesis results.

[0093] The above-described steps of fusing the first fusion feature and the predicted prosodic factor feature to obtain the second fusion feature fuse the text information of the target speech text, the prosodic information in the target speech text and the context text, as well as the predicted prosodic factor information that affects the prosodic information, and use this as the basis for speech synthesis, thereby improving the accuracy of the speech synthesis results.

[0094] Step S205: Input the second fusion feature into the trained decoder to obtain the Mel spectrum, input the Mel spectrum into the trained vocoder to obtain the sound waveform, and obtain the target synthesized speech based on the sound waveform.

[0095] In the field of speech processing, to improve the convenience of speech data computation, we need to convert between speech signals and their corresponding spectrograms, mapping the data on the spectrogram to the speech information. While typical spectrogram frequencies are linearly distributed, the human ear's perception of frequency is logarithmic—sensitive to changes in low frequencies but insensitive to changes in high frequencies. The non-linear distribution of the Mel spectrum can effectively match the human ear's frequency perception and is widely used in speech processing.

[0096] Therefore, in this embodiment, the second fusion feature is input to the trained decoder for feature mapping to obtain the Mel spectrum. Then, the Mel spectrum is input to the trained vocoder for waveform extraction to obtain the sound waveform. Finally, the corresponding target synthesized speech is obtained based on the sound waveform.

[0097] The above steps, which involve inputting the second fusion feature into a trained decoder to obtain a Mel spectrum, inputting the Mel spectrum into a trained vocoder to obtain a sound waveform, and obtaining the target synthesized speech based on the sound waveform, improve the accuracy of the target synthesized speech by using the second fusion feature, which effectively represents the text information of the target speech text, the prosodic information in the target speech text and the context text, and the information of the predictive prosodic factors that affect the prosodic information.

[0098] This invention employs a method of inputting target speech text into a trained text encoder to obtain speech text features, inputting contextual text into a trained context encoder to obtain contextual attention features, and inputting reference audio into a trained prosodic encoder to obtain prosodic features. The speech text features, contextual attention features, and prosodic features are then fused to obtain a first fused feature, effectively extracting the textual information of the text to be synthesized, as well as the prosodic information influenced by the contextual text. The first fused feature is then input into a trained prosodic factor predictor to obtain predicted prosodic factor features. The first fused feature and the predicted prosodic factor features are then fused to obtain a second fused feature, which, combined with prosodic factors such as sound duration, pitch, and energy, further enhances the richness and accuracy of prosodic information. Finally, the second fused feature is input into a trained decoder to obtain a Mel spectrum, which is then input into a trained vocoder to obtain a sound waveform. The target synthesized speech is then obtained based on the sound waveform, improving the accuracy of the target synthesized speech. In the field of medical technology, this method efficiently assists medical personnel in accurately handling a large amount of tedious and repetitive work, greatly improving their work efficiency and quality.

[0099] Corresponding to the speech synthesis method in the above embodiments, Figure 3 shows a structural block diagram of the artificial intelligence-based speech synthesis device provided in Embodiment 2 of the present invention. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0100] Referring to Figure 3, the speech synthesis device includes:

[0101] The first feature extraction module 31 is used to obtain the target speech text, the context text of the target speech text, and the reference audio. The target speech text is input into the trained text encoder to obtain speech text features, the context text is input into the trained context encoder to obtain context attention features, and the reference audio is input into the trained prosodic encoder to obtain prosodic features.

[0102] The first feature fusion module 32 is used to fuse speech text features, context attention features and prosodic features to obtain the first fused feature;

[0103] The second feature extraction module 33 is used to input the first fused feature into the trained prosodic factor predictor to obtain the predicted prosodic factor feature. The prosodic factor of the predicted prosodic factor feature includes at least one of the duration of the sound, the pitch of the sound, and the energy of the sound.

[0104] The second feature fusion module 34 is used to perform feature fusion on the first fusion feature and the predicted prosodic factor feature to obtain the second fusion feature;

[0105] The target speech synthesis module 35 is used to input the second fusion feature into the trained decoder to obtain the Mel spectrum, input the Mel spectrum into the trained vocoder to obtain the sound waveform, and obtain the target synthesized speech based on the sound waveform.

[0106] Optionally, the trained context encoder includes a trained word embedding layer, a trained bidirectional gated recurrent layer, and a trained attention mechanism layer, and the aforementioned first feature extraction module 31 includes:

[0107] The word embedding feature extraction submodule is used to input the context text into the trained word embedding layer to embed word vectors and obtain word embedding features;

[0108] The context feature association submodule is used to input word embedding features into a trained bidirectional gated recurrent layer for feature association to obtain context features;

[0109] The attention feature interaction submodule is used to input context features into the trained attention mechanism layer for feature interaction to obtain context attention features.

[0110] Optionally, the trained prosodic encoder includes a trained first convolutional layer, a trained gated recurrent layer, and a trained first fully connected layer, and the aforementioned first feature extraction module 31 includes:

[0111] The prosodic feature extraction submodule is used to input the reference audio into the trained first convolutional layer for feature extraction to obtain the first prosodic feature;

[0112] The prosodic feature association submodule is used to input the first prosodic feature into the trained gated recurrent layer for feature association to obtain the second prosodic feature;

[0113] The prosodic feature mapping submodule is used to input the second prosodic feature into the trained first fully connected layer for feature mapping to obtain the prosodic feature.

[0114] Optionally, the trained prosodic factor predictor includes a trained second convolutional layer, a trained normalization layer, and a trained second fully connected layer, and the aforementioned second feature extraction module 33 includes:

[0115] The prosodic feature extraction submodule is used to input the first fused feature into the trained second convolutional layer for feature extraction to obtain the first prosodic feature;

[0116] The prosodic feature standardization submodule is used to input the first prosodic feature into the trained standardization layer for feature standardization to obtain the second prosodic feature.

[0117] The prosodic feature mapping submodule is used to input the second prosodic feature into the trained second fully connected layer for feature mapping to obtain the predicted prosodic feature.

[0118] Optionally, the trained prosodic factor predictor includes a trained duration predictor, and the aforementioned first feature extraction module 31 includes:

[0119] The first feature extraction submodule is used to input the first fused feature into the trained duration predictor to obtain the predicted duration feature, which is used as the predicted prosodic factor feature.

[0120] Optionally, the trained prosodic factor predictor also includes a trained pitch predictor, and the aforementioned first feature extraction module 31 further includes:

[0121] The second feature extraction submodule is used to input the first fused feature into the trained pitch predictor to obtain the predicted pitch feature, and to use the predicted duration feature and the predicted pitch feature together as the predicted prosodic factor feature.

[0122] Optionally, the trained prosodic factor predictor also includes a trained energy predictor, and the aforementioned first feature extraction module 31 further includes:

[0123] The third feature extraction submodule is used to input the first fused feature into the trained energy predictor to obtain the predicted energy feature, and to use the predicted duration feature, predicted pitch feature and predicted energy feature together as the predicted prosodic factor feature.

[0124] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0125] Figure 4 is a schematic diagram of a computer device provided in Embodiment 3 of the present invention. As shown in Figure 4, the computer device of this embodiment includes: at least one processor (only one is shown in Figure 4), a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it implements the steps in any of the above-described speech synthesis method embodiments.

[0126] The computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that Figure 4 is merely an example of a computer device and does not constitute a limitation thereof. The computer device may include more or fewer components than illustrated, or a combination of certain components, or different components, such as a network interface, a display screen, and input devices.

[0127] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0128] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard disk of the computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc., equipped on the computer device. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store operating systems, applications, boot loaders, data, and other programs, such as program code of computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0129] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0130] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0133] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech synthesis method based on artificial intelligence, characterized in that, The speech synthesis method includes: acquiring target speech text, context text of the target speech text, and reference audio; inputting the target speech text into a trained text encoder to obtain speech text features; inputting the context text into a trained context encoder to obtain context attention features; inputting the reference audio into a trained prosodic encoder to obtain prosodic features; fusing the speech text features, the context attention features, and the prosodic features to obtain a first fused feature; inputting the first fused feature into a trained prosodic factor predictor to obtain predicted prosodic factor features, wherein the prosodic factors of the predicted prosodic factor features include at least one of sound duration, sound pitch, and sound energy; fusing the first fused feature and the predicted prosodic factor features to obtain a second fused feature; inputting the second fused feature into a trained decoder to obtain a Mel spectrum; inputting the Mel spectrum into a trained vocoder to obtain a sound waveform; and obtaining the target synthesized speech based on the sound waveform.

2. The speech synthesis method according to claim 1, characterized in that, The trained prosodic factor predictor includes a trained duration predictor; the step of inputting the first fused feature into the trained prosodic factor predictor to obtain predicted prosodic factor features includes: inputting the first fused feature into the trained duration predictor to obtain predicted duration features, which are used as predicted prosodic factor features.

3. The speech synthesis method according to claim 2, characterized in that, The trained prosodic factor predictor also includes a trained pitch predictor; the step of inputting the first fusion feature into the trained prosodic factor predictor to obtain predicted prosodic factor features further includes: inputting the first fusion feature into the trained pitch predictor to obtain predicted pitch features, and using the predicted duration feature and the predicted pitch feature together as predicted prosodic factor features.

4. The speech synthesis method according to claim 3, characterized in that, The trained prosodic factor predictor also includes a trained energy predictor; the step of inputting the first fusion feature into the trained prosodic factor predictor to obtain predicted prosodic factor features further includes: inputting the first fusion feature into the trained energy predictor to obtain predicted energy features, and using the predicted duration feature, the predicted pitch feature, and the predicted energy feature together as predicted prosodic factor features.

5. The speech synthesis method according to claim 1, characterized in that, The trained context encoder includes a trained word embedding layer, a trained bidirectional gated recurrent layer, and a trained attention mechanism layer; The step of inputting the context text into the trained context encoder to obtain context attention features includes: inputting the context text into the trained word embedding layer to perform word vector embedding to obtain word embedding features; inputting the word embedding features into the trained bidirectional gated recurrent layer to perform feature association to obtain context features; and inputting the context features into the trained attention mechanism layer to perform feature interaction to obtain context attention features.

6. The speech synthesis method according to claim 1, characterized in that, The trained prosodic encoder includes a trained first convolutional layer, a trained gated recurrent layer, and a trained first fully connected layer; The step of inputting the reference audio into the trained prosodic encoder to obtain prosodic features includes: inputting the reference audio into the trained first convolutional layer for feature extraction to obtain a first prosodic feature; inputting the first prosodic feature into the trained gated recurrent layer for feature association to obtain a second prosodic feature; and inputting the second prosodic feature into the trained first fully connected layer for feature mapping to obtain a prosodic feature.

7. The speech synthesis method according to claim 1, characterized in that, The trained prosodic factor predictor includes a trained second convolutional layer, a trained normalization layer, and a trained second fully connected layer. The step of inputting the first fused feature into the trained prosodic factor predictor to obtain predicted prosodic factor features includes: inputting the first fused feature into the trained second convolutional layer for feature extraction to obtain first prosodic factor features; inputting the first prosodic factor features into the trained normalization layer for feature normalization to obtain second prosodic factor features; and inputting the second prosodic factor features into the trained second fully connected layer for feature mapping to obtain predicted prosodic factor features.

8. A speech synthesis device based on artificial intelligence, characterized in that, The speech synthesis device includes: a first feature extraction module, used to acquire target speech text, context text of the target speech text, and reference audio; input the target speech text into a trained text encoder to obtain speech text features; input the context text into a trained context encoder to obtain context attention features; and input the reference audio into a trained prosodic encoder to obtain prosodic features; a first feature fusion module, used to fuse the speech text features, the context attention features, and the prosodic features to obtain a first fused feature; a second feature extraction module, used to input the first fused feature into a trained prosodic factor predictor to obtain predicted prosodic factor features, wherein the prosodic factors of the predicted prosodic factor features include at least one of sound duration, sound pitch, and sound energy; a second feature fusion module, used to fuse the first fused feature and the predicted prosodic factor features to obtain a second fused feature; and a target speech synthesis module, used to input the second fused feature into a trained decoder to obtain a Mel spectrum; input the Mel spectrum into a trained vocoder to obtain a sound waveform; and obtain target synthesized speech based on the sound waveform.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method as described in any one of claims 1 to 7.

Images