Generating voice data using artificial intelligence techniques

Abstract

Methods, systems, and computer program products for generating speech data using artificial intelligence techniques are provided herein. A computer-implemented method includes implementing one or more artificial intelligence techniques in association with one or more speech synthesis tasks; generating at least one data sequence including one or more of phoneme data and rhythm data by processing at least one previously generated data sequence using one or more artificial intelligence techniques in the plurality of contiguous portions; and generating speech data corresponding to at least a portion of the data sequence by processing the at least a portion of the data sequence using at least one artificial intelligence-based speech synthesis model.

Description

BACKGROUND

[0001] The present application relates generally to information technology, and more particularly to language and speech processing. More specifically, there arises a situation where it is desirable to predict a text output of a word or token given a prior text, and if such output is to be used in a context related to a conversation, the text needs to be converted to speech data. However, conventional speech synthesis techniques include limitations such as significant latency issues, accuracy issues, and inability to adequately capture style and emotion in speech data. SUMMARY

[0002] In at least one embodiment, techniques are provided for generating speech data using artificial intelligence techniques.

[0003] An example computer-implemented method includes implementing one or more artificial intelligence techniques in association with one or more speech synthesis tasks, and generating at least one sequence of data including one or more of phoneme data and prosody data in a plurality of successive portions by processing at least one previously generated sequence of data using the one or more artificial intelligence techniques. Additionally, the method further includes generating speech data corresponding to at least a portion of the sequence of data by processing the at least a portion of the sequence of data using at least one artificial intelligence-based speech synthesis model.

[0004] At least one embodiment can include combining at least one language model (LM), at least one text-to-speech front-end (TTS-FE) model, and at least one text-to-speech prosody (TTS-P) model. Additionally or alternatively, one or more embodiments can include using a language model text-to-speech algorithm adapter to convert at least a portion of output from the at least one LM into phoneme data and prosody data to be used as input to one or more of the at least one TTS-FE model and the at least one TTS-P model.

[0005] Another embodiment of the application, or of elements thereof, can be implemented in a computer program product, which can include one or more computer program elements. When the computer program elements are loaded into and executed by a computer, they cause the computer to perform a series of operations to provide features described herein. Alternatively, this application can be implemented in hardware, or a combination of hardware and software. Further, the embodiments disclosed herein can be implemented in a variety of computing systems using a variety of hardware configurations and computing devices.

[0006] Illustrative embodiments can provide significant advantages over traditional speech synthesis techniques. For example, in one or more embodiments, such as those mentioned above, by using artificial intelligence techniques to generate speech data in association with phoneme data and / or prosody data, issues associated with latency problems, accuracy problems, and an inability to adequately capture style and emotion in speech data are overcome.

[0007] These and other objects, features, and advantages of the present application will become apparent with reference to the detailed description of illustrative embodiments thereof, which describe an example of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a diagram illustrating training of an LM using phonetic transcripts according to example embodiments of the present application;

[0009] Figure 2 is a diagram illustrating generation of speech data using a text-to-speech (TTS) system and an LM trained with phonetic information according to example embodiments of the present application;

[0010] Figure 3 is a diagram illustrating training of an LM using phonetic transcripts and prosody features according to example embodiments of the present application;

[0011] Figure 4 is a diagram illustrating generation of speech data using a TTS system and an LM trained with phonetic information and prosody features according to example embodiments of the present application;

[0012] Figure 5 is a diagram illustrating fine-tuning of an LM using speech data according to example embodiments of the present application;

[0013] Figure 6 is a diagram illustrating a language model-TTS (LM-TTS) adapter architecture according to example embodiments of the present application;

[0014] Figure 7 is a diagram illustrating example LM-TTS adapter output according to example embodiments of the present application;

[0015] Figure 8 is a flow diagram illustrating techniques according to example embodiments of the present application; and

[0016] Figure 9 is a diagram illustrating a computing environment in which at least one embodiment of the present application can be implemented. DETAILED DESCRIPTION

[0017] As described herein, at least one embodiment includes generating speech data using artificial intelligence techniques. In one or more embodiments, one or more LMs are trained to output the next text token in a sequence given some text input. Additionally or alternatively, one or more LMs can be trained to output the next text token based on one or more previously generated tokens and optionally based on a query. Such output text is converted into speech data using, for example, at least one speech synthesis system when used, for example, in association with a voice conversational context and / or implementation.

[0018] Further, as used herein, a generative language model (G-LM) refers to an autoregressive language model or autoregressive text prediction model that sequentially generates words (e.g., a fixed number of words at a time) given a sequence of previously synthesized words (optionally, starting with a text query), and can also output a sequence of related internal states. Additionally, as used herein, a generative text-to-speech frontend (G-TTS-FE) model refers to a neural model that sequentially generates a sequence of TTS-FE symbols (e.g., symbols suitable for a speech synthesis process (e.g., a fixed number of symbols at a time)) from plain text and / or annotated text (e.g., a sequence of words), where the sequence of symbols suitable for speech synthesis includes at least one phoneme sequence (e.g., phonemes, dictionary stress, etc.), optionally includes phrase type information, and optionally includes one or more annotations such as part of speech, word emphasis, etc.

[0019] Further, as used herein, a generative text-to-speech prosody (G-TTS-P) model refers to a neural model that sequentially generates a sequence of TTS-Ps from plain text and / or annotated text, where the sequence of TTS-Ps includes prosodic feature vectors (e.g., a fixed number of prosodic feature vectors at a time) that guide at least one subsequent speech synthesis system. Such prosodic feature vectors can include, for example, global and / or local assessments of phone rate, pitch, and volume. Additionally, such prosodic feature vectors can be derived, for example, from one or more statistical measures taken over one or more hierarchical time intervals, and / or normalized to become one or more speaker-agnostic features. Also as used herein, a generative speech synthesis (G-SS) model refers to a neural model that sequentially generates at least one output speech waveform from at least one sequence of TTS-FE symbols and at least one sequence of TTS-P feature vectors.

[0020] Accordingly, at least one embodiment can include generating and / or implementing a combined G-LM, G-TTS-FE, and G-TTS-P model that sequentially generates TTS-FE and TTS-P outputs, and optionally one or more related text outputs, with the combined model followed by a G-SS model. Additionally or alternatively, one or more embodiments can include generating and / or implementing a combined G-TTS-FE and G-TTS-P model that sequentially generates TTS-FE and TTS-P outputs, with the combined model preceded by a G-LM model, and using its internal state sequence as input in addition to the text sequence, and followed by a G-SS model. Further, at least one embodiment can include generating and / or implementing a combined G-LM, G-TTS-FE, G-TTS-P, and G-SS model that performs the entire end-to-end speech synthesis task.

[0021] Further, as described in further detail herein, at least one embodiment includes using the output of one or more TTS systems along with at least one LM (e.g., at least one G-LM) to generate speech data. Such embodiments include training at least one LM to produce one or more types of output, where such output can include phoneme transcription data, one or more phrase types and part-of-speech tags, word and / or syllable stress information, pause information and / or phrase break information, prosody information (e.g., prosodic labels useful for real-time speech synthesis), etc. In one or more embodiments, such output can be used as direct input to a TTS synthesis model, enabling the TTS synthesis model to begin synthesis with minimal delay, and produce more natural speech output that correctly conveys the meaning of the input.

[0022] As described in further detail herein, at least one embodiment includes modifying a LM, e.g., by training the LM to output phonemes and linguistic information in parallel with outputting words. In such embodiments, such LM training can include using original LM training text, and extracting one or more phonemes and linguistic information (e.g., phrase type information) from it using, e.g., at least one TTS language frontend, grapheme-to-phoneme or other similar language analysis tools.

[0023] Additionally, one or more embodiments include modifying the LM, e.g., by training the LM to produce prosodic information such as pitch, phoneme duration, phrase break information, etc. In such embodiments, such LM training can include feeding the original LM training text to a TTS system and / or a TTS prosody prediction system (e.g., a G-TTS-P model and / or a G-TTS-FE model as independent or internal TTS components) and extracting prosodic information (e.g., pitch and energy curves, as well as phoneme durations) therefrom. One or more TTS systems may, for example, include a G-TTS-P model as an internal component of the TTS system, while other TTS systems (e.g., end-to-end TTS systems) do not include such a model. In such cases, speech data can be synthesized and the desired prosodic features can be extracted from the waveforms.

[0024] Further, at least one embodiment includes modifying at least one TTS system to use information generated by at least one LM (e.g., a G-LM) according to at least a portion of the techniques detailed herein as input, rather than just text input.

[0025] As noted above, one or more embodiments include training a LM to produce extended phoneme transcription information and / or hybrid output, where, for example, a query is given as plain text but the response is generated at least partially in phoneme output. To train the LM accordingly, at least one embodiment can include using at least one existing LM training text corpus and converting at least a portion of such text to extended phoneme script information using at least one TTS system language front end. Such resulting script information can then be used to train the LM.

[0026] One or more embodiments can also include incorporating additional information to such script information by processing at least a portion of the script information using a TTS model. In association with training at least one LM, such additional information can produce enhanced intonation, phoneme duration, pitch information, pause information, etc., which are helpful to the relevant TTS system. By way of example only, such additional information can include prosodic information, which can convey speaker-agnostic speech synthesis features, e.g., hierarchical prosody control (HPC) features. Other types of information can include, for example, word stress, emotion (e.g., happy, apologetic, etc.), and speaking style (e.g., conversational, announcement, reading, etc.). Further, in association with incorporating additional information into script information, such action does not require the use and / or obtaining of additional data, as the additional information can be generated from the original training text corpus.

[0027] In one or more embodiments, a TTS system using such extended speech script information and additional and / or prosodic information can use the output of the LM as direct input, which can reduce latency because the TTS system will not need long look-ahead times to understand the language context and present speech with natural prosody.

[0028] Furthermore, at least one embodiment may include using speech data to fine-tune and / or enhance the LM. This embodiment includes extracting phoneme and intonation information from the speech data and using at least a portion of this extracted information to train the LM. Such speech data may include, for example, a large speech corpus from many speakers, which can facilitate general intonation improvement, and / or speech data from a single speaker for adaptation to a specific voice or style (e.g., conversational voice style). Alternatively or additionally, for data from multiple speakers, one or more speaker-agnostic features (e.g., HPC features) may be utilized and / or required.

[0029] As detailed herein, at least one embodiment can provide beneficial effects, such as reducing latency and increasing accuracy in LM and TTS system implementations. In another embodiment, a pre-trained LM is utilized without modification. In such an embodiment, an LM-TTS adapter is created and / or implemented, wherein the LM-TTS adapter takes the pre-trained LM output and / or its internal state parameters as input and outputs at least one phoneme and prosodic information required for a corresponding TTS. In one example embodiment, the LM-TTS adapter uses an encoder-decoder architecture. In such an embodiment, the encoder generates an encoding vector for each input word and may have adjustable word look-ahead. The decoder takes the encoder output and one or more previous outputs and produces the phoneme and prosodic information for the current word. Alternatively, one or more embodiments may include implementing separate phoneme and prosodic decoders.

[0030] Figure 1 This is a schematic diagram illustrating the use of phoneme transcription to train an LM according to an exemplary embodiment of the present invention. As an illustration, Figure 1 The text LM 102 is described as being converted into a phoneme LM 104 using a training corpus 105 of phoneme data. Such a trained phoneme LM can generate phoneme transcription data instead of text data and / or, in addition to text data, generate phonemes, stress information, phrase type information, part-of-speech information, interruption information, etc. In one or more embodiments, the text LM 102 may include a pre-trained text LM, and the training corpus 105 for generating phoneme data may include converting at least a portion of the training corpus 103 of text data into phoneme data using a language front-end 106 (e.g., one or more text processing programs) from a TTS system.

[0031] In at least one embodiment, such as Figure 1 As shown optionally, training speech data (e.g., spoken text) 109 obtained using Automatic Speech Recognition (ASR) technology 108 can also be incorporated into the training corpus 105 along with text data provided by the language front-end 106. Thus, as trained, the phoneme LM 104 can process the input text data (e.g., a question) and generate a response (e.g., an answer to a question) in phonemes. One or more embodiments may also include incorporating one or more external tags, such as emotion or word emphasis.

[0032] Figure 2 This is a schematic diagram illustrating the use of a TTS system and an LM trained with phoneme information to generate speech data according to an exemplary embodiment of the present invention. For illustration, Figure 2 The phoneme LM 204 is described (e.g., such as Figure 1 The trained phoneme LM (as depicted in the text) processes text input (e.g., a query) and generates phoneme output (e.g., a response to a question). This phoneme output (e.g., a phoneme script) is then provided as input to a TTS system 210 (e.g., a simplified TTS system without a language front-end and / or trained by an LM (such as...). Figure 3 The phoneme LM 204 (described in the image) is a speech synthesis system controlled by a selected prosodic feature set, which processes the input and generates corresponding speech data. According to one or more embodiments, the phoneme LM 204 has a considerable context with the required annotations for producing the phoneme output, and the phoneme LM 204 can be trained to correctly handle isomorphic graph and / or text normalization tasks.

[0033] Figure 3 This is a schematic diagram illustrating the training of an LM using phoneme transcription and prosodic features according to an exemplary embodiment of the present invention. As an illustration, Figure 3 The text LM 302 is modified into phoneme LM 304 using a training corpus 307 containing phoneme and prosodic data. This trained phoneme LM 304 generates enhanced phoneme transcription data with associated prosodic information, rather than the text data and / or anything other than text data. Generating the training corpus 307 may include converting at least a portion of the training corpus 303 containing text data into phoneme data using a TTS language front-end 306. Also... Figure 3As shown, one or more embodiments include enhancing the training corpus 307 by adding information about prosody and / or tone obtained from prosodic model 312. The prosodic information incorporated into the training corpus 307 may include, for example, one or more prosodic cue labels (e.g., cue for longer syllables, shorter syllables, higher pitch, lower pitch, etc.), one or more hierarchical labels for speech segments (e.g., sentence, word, and phoneme modifiers), full pitch curves, and phoneme durations, etc. In addition to using prosodic model 312, such prosodic information can be generated, for example, by applying a TTS system to a training corpus 303 of text data (e.g., offline) and / or by real speech data (e.g., using prosodic model 312).

[0034] Figure 4 This is a schematic diagram illustrating the generation of speech data using a TTS system and an LM trained with phoneme information and prosodic features according to an exemplary embodiment of the present invention. As an illustration, Figure 4 The phonemes and prosody of LM 404 are described (e.g., such as...). Figure 3 The trained phoneme and prosody LM (as depicted in the text) processes text input (e.g., a query) and generates phoneme and prosody output (e.g., a response to the query). This output is then provided as input to a TTS system 410 (e.g., a low-latency TTS system), which processes the input and generates corresponding speech data. In at least one embodiment, the TTS system 410 is modified to use and / or process the phoneme script and associated prosody information (as generated by the phoneme and prosody LM 404) as input. This can, for example, reduce the look-ahead of the TTS system 410 (e.g., reduce the look-ahead from several words to several phonemes) and provide a system that can generate speech data approximately as quickly as the output data that the LM can generate to be fed to the TTS system.

[0035] Figure 5 This is a schematic diagram illustrating the use of voice data to fine-tune the LM according to an exemplary embodiment of the present invention. As an illustration, Figure 5 The text LM 502 is modified into phoneme LM 504 using a training corpus 505 of phoneme data, wherein such trained phoneme LM 504 can generate enhanced speech transcription data, rather than text data and / or data other than text data. Similarly, as... Figure 5As shown, one or more embodiments include enhancing a training corpus 505, and subsequently fine-tuning a phoneme LM 504 using training speech data 509 (e.g., real speech data), which is processed using an ASR 508 and a prosodic feature extractor 516. The ASR 508 is used to extract information such as phonemes and phoneme duration, and the prosodic feature extractor 516 can extract information such as pitch curves and energy. Fine-tuning the phoneme LM 504 using speech data can improve the quality and naturalness of the phoneme LM 504 output and can facilitate the adaptation of the phoneme LM 504 to aspects such as, for example, specific speaker, specific speaking style (e.g., conversational speech style), emotion, and articulation.

[0036] Alternatively or concurrently, at least one embodiment includes connecting and / or using at least one LM and at least one TTS system, wherein the at least one LM-TTS adapter acquires the LM output and its internal state, and converts such output into one or more phonemes and one or more prosodic controls, which can be used as input to the TTS system. In such embodiments, the LM-TTS adapter model can leverage the hidden states of the language model to improve accuracy and support conversion in parallel with LM text generation.

[0037] Figure 6 This is a schematic diagram illustrating an LM-TTS adapter architecture according to an exemplary embodiment of the present invention. For illustration, Figure 6 An example LM-TTS adapter model representing an extended version of the transformer encoder-decoder architecture is depicted. Specifically, Figure 6 The LM 602 is described as processing text queries to generate one or more internal state vectors, word embedding vectors, and language model lexical units 620 (e.g., one or more text word fragments). Furthermore, Figure 6 The LM-TTS adapter 660 is described, which includes an encoder 662 and a decoder 664.

[0038] Encoder 662 processes input including at least a portion of language model lexical 620, which is combined with internal state and context embedding vectors from LM 602. Encoder 662 outputs an embedding vector for each word. In one or more embodiments, at least a portion of the language model lexical 620 and semantic information associated with the embeddings may be obtained from one or more internal layers of LM 602 (e.g., one or more deep layers and / or one or more shallow layers) and fed into encoder 662. By way of example only, in at least one embodiment, encoder 662 includes two transformer layers with 512 embedding dimensions and eight attention heads.

[0039] Additionally, in one or more embodiments, encoder 662 does not focus on future words, which is a constraint that ensures the prediction of a word's phonemes remains unchanged for future words. In such embodiments, this constraint can also be relaxed and can be added to and / or incorporated into a fixed lookahead, thereby facilitating a trade-off between delay and increased context.

[0040] For example Figure 6 As depicted, decoder 664 processes at least a portion of the output generated by encoder 662 and produces and / or outputs one or more phonemes and prosodic information (e.g., one or more prosodic features). Furthermore, in one or more embodiments, each phoneme output by decoder 664 is selected from at least one phoneme vocabulary, and the prosodic information may include one or more normalized prosodic observations. Each prosodic observation may include, for example, a normalized linear combination of statistical measurements evaluating a specific prosodic metric (e.g., pitch [Hz], rhythm [phoneme duration], loudness [dB], etc.) over a predetermined time period. In such embodiments, implementing decoder 664 may include deploying a set of hierarchical aggregations (e.g., sentence-level aggregations, word-level aggregations, etc.).

[0041] In at least one embodiment, decoder 664 predicts one or more phoneme and prosodic information via autoregressive prediction. In such an embodiment, decoder 664 takes encoder output (e.g., word embedding vectors) and previously generated phoneme and prosodic outputs as input and predicts the next phoneme and prosodic output, an example of which is... Figure 7 As shown in the image.

[0042] Figure 7 This is a schematic diagram illustrating an exemplary modified LM or LM-TTS adapter output 700 according to an example embodiment of the present invention. As an illustration, Figure 7 An exemplary modified LM or LM-TTS adapter output 700 is depicted in tabular form, which includes information related to decoding steps, LM text output, phoneme output, and word-level and sentence-level HPC parameters.

[0043] Refer again Figure 6In one or more example embodiments, decoder 664 may include four transformer layers with 512 embedding dimensions and eight attention heads. While LM 602 generates text, LM-TTS adapter 660 operates in parallel. By way of example only, in at least one embodiment, each time LM 602 completes the generation of a word, encoder 662 runs on the LM output and subsequently passes it multiple times through decoder 664, which autoregressively generates a relevant sequence of phonemes and prosodic outputs until decoder 664 predicts the end-word lexical. After decoder 664 stops, its output is sent to TTS system 610 to be synthesized into speech data.

[0044] Figure 8 This is a flowchart illustrating a technique according to an embodiment of the present invention. Step 802 includes implementing one or more artificial intelligence techniques (e.g., at least one artificial neural network (ANN) module) in association with one or more speech synthesis tasks. In at least one embodiment, implementing one or more artificial intelligence techniques includes combining at least one LM, at least one TTS-FE model, and at least one TTS-P model. Additionally or alternatively, implementing one or more artificial intelligence techniques may include using a language model text-to-speech algorithm adapter to convert at least a portion of the output from at least one LM into phoneme data and prosodic data to be used as input to one or more of at least one TTS-FE model and at least one TTS-P model. In such an embodiment, implementing one or more artificial intelligence techniques may include using a language model text-to-speech algorithm adapter in conjunction with generating one or more HPC features, the one or more HPC features being derived from one or more statistical measurements obtained at one or more hierarchical time intervals and normalized to represent one or more speaker-agnostic features, wherein at least a portion of the one or more HPC features includes one or more local evaluations and one or more global evaluations of at least one of phoneme rate, pitch, and / or volume.

[0045] Furthermore, in one or more embodiments, implementing one or more artificial intelligence techniques includes using one or more phoneme data items to modify at least a portion of at least one LM. In such embodiments, implementing one or more artificial intelligence techniques may include using at least one TTS-FE model to extract one or more phoneme data items from at least one set of training text data.

[0046] Alternatively or concurrently, implementing one or more artificial intelligence techniques may include using one or more prosodic data items (e.g., pitch-related information, phoneme duration-related information, and / or phrase break-related information) to modify at least a portion of at least one LM. In such embodiments, implementing one or more artificial intelligence techniques may include using at least one TTS-P model to extract one or more prosodic data items from at least one set of training text data.

[0047] Furthermore, in at least one embodiment, implementing one or more artificial intelligence techniques includes using one or more speech data items to modify at least a portion of at least one LM in association with at least one automated speech recognition technique and one or more prosodic feature extraction techniques.

[0048] Step 804 includes generating at least one data sequence comprising one or more of phoneme data and prosodic data in multiple consecutive segments by processing at least one previously generated data sequence (and optionally, related text queries) using one or more artificial intelligence techniques. In one or more embodiments, generating at least one data sequence includes generating one or more phonemes in combination with one or more output words associated with said at least one previously generated data sequence, and generating one or more prosodic information items. In such embodiments, generating one or more prosodic information items may include generating one or more hierarchical prosodic control (HPC) features derived from one or more statistical measurements obtained at one or more hierarchical time intervals and normalized to represent one or more speaker-agnostic features, wherein at least a portion of the one or more HPC features includes at least one or more global evaluations and one or more local evaluations of at least one of phoneme rate, pitch, and volume.

[0049] Alternatively, generating at least one data sequence comprising one or more of phoneme data and prosodic data in multiple consecutive portions may include generating the at least one data sequence, each time comprising one or more speech vectors and one or more prosodic vectors.

[0050] Step 806 includes generating speech data corresponding to at least a portion of the data sequence by processing at least a portion of the data sequence using at least one artificial intelligence-based speech synthesis model. In at least one embodiment, generating speech data includes generating one or more speech waveforms corresponding to at least a portion of the data sequence.

[0051] Figure 8 The techniques described may also include automatically training at least a portion of one or more artificial intelligence techniques using at least a portion of the generated speech data. Additionally, in one or more embodiments, implementation... Figure 8The software based on the technology described herein can be provided as a service in a cloud environment.

[0052] The illustrative embodiments described above offer significant advantages over conventional methods. For example, some embodiments are configured to use artificial intelligence techniques in association with phoneme data and / or prosodic data to generate speech data. These and other embodiments can effectively overcome problems related to latency, accuracy, and the inability to adequately capture the style and emotion associated with the speech data.

[0053] It should be understood that some embodiments described herein utilize one or more artificial intelligence models. It should be understood that the term "model" as used herein is intended to be interpreted broadly and may include, for example, a set of executable instructions for generating computer-implemented recommendations and / or predictions. For example, one or more of the models described herein can be trained to generate recommendations and / or predictions based on input text data, phoneme data, and / or prosodic information, and such recommendations and / or predictions can be used to initiate one or more automated actions (e.g., automatically generating speech data related to one or more TTS systems, automatically training one or more artificial intelligence techniques (e.g., one or more LMs), etc.).

[0054] As described in this article, Figure 8 The techniques described herein may also include providing a system comprising different software modules, each contained on a tangible computer-readable and recordable storage medium. For example, all modules (or any subset thereof) may be on the same medium, or each may be on a different medium. Modules may include any or all of the components shown in the figures and / or described herein. In embodiments of the invention, modules may, for example, run on a hardware processor. The method steps can then be performed using the different software modules of a system executed on a hardware processor as described above. Furthermore, the computer program product may include a tangible computer-readable and recordable storage medium having code adapted to be executed to perform at least one method step described herein, including providing the different software modules to the system.

[0055] in addition, Figure 8 The techniques described herein can be implemented via a computer program product that may include computer-usable program code stored in a computer-readable storage medium in a data processing system, wherein the computer-usable program code is downloaded from a remote data processing system via a network. Furthermore, in embodiments of the invention, the computer program product may include computer-usable program code stored in a computer-readable storage medium in a server data processing system, wherein the computer-usable program code is downloaded to a remote data processing system via a network for use in a computer-readable storage medium having a remote system.

[0056] Embodiments of the present invention or elements thereof may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and configured to perform exemplary method steps.

[0057] Various aspects of this disclosure are described by narrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). Regarding any flowchart, depending on the technology involved, operations may be performed in a different order than that shown in a given flowchart. For example, again according to the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0058] Computer Program Product Embodiment (“CPP Embodiment” or “CPP”) is a term used in this disclosure to describe any collection of one or more storage media (also referred to as “media”) collectively included in a collection of one or more storage devices, the collection of one or more storage devices collectively including machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A “storage device” is any tangible device capable of holding and storing instructions used by a computer processor. Without limitation, a computer-readable storage medium can be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punch cards or pits / platforms formed in the main surface of the disk), or any suitable combination of the foregoing. Computer-readable storage media, as used in this disclosure, should not be construed as storing transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As those skilled in the art will understand, data is typically moved at certain incidental points in time during the normal operation of the storage device, such as during access, defragmentation, or garbage collection; however, this does not make the storage device transient, because the data is not transient when it is stored.

[0059] The computing environment 900 includes examples of an environment for executing at least some of the computer code involved in performing the methods of the present invention, such as enhanced speech data generation code 926. In addition to code 926, the computing environment 900 includes, for example, a computer 901, a wide area network (WAN) 902, an end-user equipment (EUD) 903, a remote server 904, a public cloud 905, and a private cloud 906. In this embodiment, the computer 901 includes a processor set 910 (including processing circuitry 920 and a cache 921), a communication structure 911, volatile memory 912, persistent storage 913 (including an operating system 922 and code 926 as described above), a peripheral device set 914 (including a user interface (UI) device set 923, storage 924, and an Internet of Things (IoT) sensor set 925), and a network module 915. The remote server 904 includes a remote database 930. Public cloud 905 includes gateway 940, cloud coordination module 941, host physical machine set 942, virtual machine set 943, and container set 944.

[0060] Computer 901 can take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or to be developed in the future capable of running programs, accessing networks, or querying databases such as remote database 930. As is well known in the field of computer technology, and depending on that technology, the execution of computer-implemented methods can be distributed among multiple computers and / or multiple locations. On the other hand, in this presentation of computing environment 900, the detailed discussion focuses on a single computer, specifically computer 901, to keep the presentation as simple as possible. Computer 901 can reside in the cloud, even... Figure 9 The document does not show that it is in the cloud; on the other hand, computer 901 does not need to be in the cloud unless to the extent that can be definitively indicated.

[0061] Processor set 910 includes one or more computer processors of any type now known or to be developed in the future. Processing circuitry 920 may be distributed across multiple packages, such as multiple cooperating integrated circuit chips. Processing circuitry 920 may implement multiple processor threads and / or multiple processor cores. Cache 921 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by the threads or cores running on processor set 910. Cache memory is typically organized into multiple levels based on its relative proximity to the processing circuitry. Alternatively, some or all of the cache in the processor set may be located “off-chip.” In some computing environments, processor set 910 may be designed to work with qubits and perform quantum computing.

[0062] Computer-readable program instructions are typically loaded onto computer 901 to cause processor set 910 of computer 901 to perform a series of operational steps to implement a computer-implemented method, such that the instructions thus executed instantiate the method specified in the flowchart and / or the description of the computer-implemented method included in this document (collectively, the “method of the invention”). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 921 and other storage media discussed below. The program instructions and associated data are accessed by processor set 910 to control and direct the execution of the method of the invention. In computing environment 900, at least some instructions for performing the method of the invention may be stored in code 926 in persistent storage 913.

[0063] The communication structure 911 is a signal transmission path that allows various components of the computer 901 to communicate with each other. Typically, this structure consists of switches and conductive paths, such as switches and conductive paths that form buses, bridges, physical input / output ports, etc. Other types of signal communication paths can be used, such as fiber optic communication paths and / or wireless communication paths.

[0064] Volatile memory 912 is any type of volatile memory now known or to be developed in the future. Examples include dynamic type RAM or static type RAM. Typically, volatile memory 912 is characterized by random access, but this is not required unless explicitly stated otherwise. In computer 901, volatile memory 912 is located in a single package and is internal to computer 901; however, alternatively or additionally, volatile memory may be distributed across multiple packages and / or located externally relative to computer 901.

[0065] Persistent storage 913 is any form of non-volatile storage for a computer, now known or to be developed in the future. The non-volatility of this storage means that the stored data is retained regardless of whether power is supplied to the computer 901 and / or directly to the persistent storage 913. Persistent storage 913 may be ROM, but typically at least a portion of persistent storage allows for data writing, data deletion, and data rewriting. Some common forms of persistent storage include hard disks and solid-state storage devices. Operating system 922 may take several forms, such as various known proprietary operating systems or operating systems employing an open-source portable operating system interface type with a kernel. The code included in code 926 typically includes at least some of the computer code involved in performing the methods of the present invention.

[0066] Peripheral device set 914 includes a collection of peripheral devices for computer 901. Data communication connections between peripheral devices and other components of computer 901 can be implemented in various ways, such as Bluetooth connections, near field communication (NFC) connections, connections made by cables (such as Universal Serial Bus (USB) type cables), plug-in connections (e.g., secure digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various embodiments, UI device set 923 may include components such as displays, speakers, microphones, wearable devices (such as glasses and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Memory 924 is an external storage device, such as an external hard drive, or a pluggable storage device, such as an SD card. Memory 924 can be persistent and / or volatile. In some embodiments, memory 924 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 901 requires substantial storage (e.g., where computer 901 locally stores and manages a large database), this storage can be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 925 consists of sensors that can be used in IoT applications. For example, one sensor could be a thermometer, while another could be a motion detector.

[0067] Network module 915 is a collection of computer software, hardware, and firmware that allows computer 901 to communicate with other computers via WAN 902. Network module 915 may include hardware such as a modem or Wi-Fi transceiver, software for packetizing and / or depacketizing data transmitted over the communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control and network forwarding functions of network module 915 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN), the control and forwarding functions of network module 915 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for performing the methods of the present invention can typically be downloaded to computer 901 from an external computer or external storage device via a network adapter card or network interface included in network module 915.

[0068] A WAN 902 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances using any technology known now or developed in the future for transmitting computer data. In some embodiments, a WAN 902 may be replaced by and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area such as a Wi-Fi network. WANs and / or LANs typically include computer hardware such as copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0069] End-user equipment 903 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 901) and can take any of the forms discussed above in conjunction with computer 901. EUD 903 typically receives helpful and useful data from the operation of computer 901. For example, assuming computer 901 is designed to provide recommendations to the end user, these recommendations are typically transmitted from network module 915 of computer 901 to EUD 903 via WAN 902. In this way, EUD 903 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 903 can be a client device, such as a thin client, a heavy client, a mainframe computer, a desktop computer, etc.

[0070] Remote server 904 is any computer system that provides at least some data and / or functionality to computer 901. Remote server 904 can be controlled and used by the same entity operating computer 901. Remote server 904 represents a machine that collects and stores useful and useful data used by other computers such as computer 901. For example, if computer 901 is designed and programmed to provide recommendations based on historical data, that historical data can be provided to computer 901 from a remote database 930 of remote server 904.

[0071] Public cloud 905 is any computer system that can be used by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities (especially data storage (cloud storage) and computing power) without direct, active management by the user. Cloud computing typically leverages resource sharing to achieve scalability consistency and economy. Direct and active management of the computing resources of public cloud 905 is performed by the computer hardware and / or software of cloud orchestration module 941. The computing resources provided by public cloud 905 are typically implemented by virtual computing environments running on various computers constituting host physical set 942, which is the entire domain of physical computers in and / or available to public cloud 905. Virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set 943 and / or containers from container set 944. It should be understood that these VCEs can be stored as images and can be transferred between various physical machine hosts as images or after instantiation of VCEs. The cloud orchestration module 941 manages the delivery and storage of images, deploys new instantiations of VCE, and manages the instantiation of VCE deployment activities. The gateway 940 is a collection of computer software, hardware, and firmware that allows the public cloud 905 to communicate via WAN 902.

[0072] Now, some further explanation of VCE will be provided. VCE can be stored as an "image." New active instances of VCE can be instantiated from this image. Two common types of VCE are virtual machines and containers. Containers are VCEs that use operating system-level virtualization. This refers to an operating system feature where the kernel allows multiple isolated user-space instances, called containers, to exist. From the perspective of the programs running within them, these isolated user-space instances typically appear as actual computers. Computer programs running on a regular operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running within containers can only use the contents of the container and the devices allocated to the container; this is a characteristic known as containerization.

[0073] Private cloud 906 is similar to public cloud 905, except that computing resources are available only to a single enterprise. While private cloud 906 is depicted as communicating with WAN 902, in other embodiments, private cloud can be completely disconnected from the internet and accessed only via a local / private network. Hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types) typically implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardization or proprietary technology that enables coordination, management, and / or data / application portability across the multiple component clouds. In this embodiment, public cloud 905 and private cloud 906 are both part of a larger hybrid cloud.

[0074] In computing environment 900, computer 901 is shown connected to the Internet (see WAN 902). However, in many embodiments of the invention, computer 901 will be isolated from communications via a communication network and will not be connected to the Internet, operating as a stand-alone computer. In these embodiments, network module 915 of computer 901 may not be necessary or even desired to ensure isolation and prevent external communications from entering computer 901. Stand-alone computer embodiments are potentially advantageous at least in some applications of the invention because they are generally more secure. In other embodiments, computer 901 is connected to a secure WAN or secure LAN instead of WAN 902 and / or the Internet. In these network-connected (i.e., non-stand-alone) embodiments, system designers may want to take appropriate security measures, now known or in the future, to mitigate the risks of incoming network communications without leading to security vulnerabilities.

[0075] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of another feature, step, operation, element, component, and / or combination thereof.

[0076] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or technical improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A system comprising: a memory configured to store program instructions; and a processor operatively coupled to the memory to execute the program instructions to: implement one or more artificial intelligence techniques in association with one or more speech synthesis tasks; generate at least one data sequence comprising one or more of phoneme data and prosody data by processing at least one previously generated data sequence in a plurality of successive portions using the one or more artificial intelligence techniques; and generate speech data corresponding to at least a portion of the data sequence by processing the at least a portion of the data sequence using at least one artificial intelligence-based speech synthesis model.

2. The system of claim 1, wherein generating speech data comprises generating one or more speech waveforms corresponding to at least a portion of the data sequence. combining at least one language model (LM), at least one text-to-speech frontend (TTS-FE) model, and at least one text-to-speech prosody (TTS-P) model.

3. The system of claim 1, wherein implementing one or more artificial intelligence techniques comprises: using a language model text-to-speech algorithm adapter to convert at least a portion of output from the at least one LM into phoneme data and prosody data to be used as input to one or more of the at least one TTS-FE model and the at least one TTS-P model.

4. The system of claim 1, wherein implementing one or more artificial intelligence techniques comprises: using the language model text-to-speech algorithm adapter in conjunction with generating one or more hierarchical prosody control (HPC) features derived from one or more statistical measures taken over one or more hierarchical time intervals and normalized to represent one or more speaker-independent features, wherein at least a portion of the one or more HPC features comprise at least one of one or more global assessments and one or more local assessments of at least one of phoneme rate, pitch, and volume.

5. The system of claim 4, wherein implementing one or more artificial intelligence techniques comprises:

6. The system of claim 1, wherein implementing one or more artificial intelligence techniques comprises: using one or more phoneme data items to modify at least a portion of at least one LM; and using at least one TTS-FE model to extract the one or more phoneme data items from at least one set of training text data.

7. The system of claim 1, wherein implementing one or more artificial intelligence techniques comprises: using one or more prosody data items to modify at least a portion of at least one LM; and using at least one TTS-P model to extract the one or more prosody data items from at least one set of training text data. using one or more speech data items to modify at least a portion of at least one LM in association with at least one automated speech recognition technique and one or more prosody feature extraction techniques.

8. The system of claim 1, wherein implementing one or more artificial intelligence techniques comprises: and generating one or more phonemes and one or more prosody information items in conjunction with one or more output words related to the at least one previously generated data sequence.

9. The system of claim 1, wherein generating at least one data sequence comprises: ​ 10. The system of claim 9, wherein generating one or more prosodic information items comprises: generating one or more HPC features derived from one or more statistical measures taken over one or more hierarchical time intervals and normalized to represent one or more speaker-independent features, wherein at least a portion of the one or more HPC features include at least one of one or more global assessments and one or more local assessments of at least one of a phonetic rate, a pitch, and a volume.

11. The system of claim 1, wherein generating, in a plurality of successive portions, at least one sequence of data comprising one or more of phoneme data and prosody data comprises: generating the at least one data sequence, one or more phonetic vectors and one or more prosodic vectors at a time.

12. The system of claim 1, wherein the processor is further operatively coupled to the memory to execute the program instructions to: automatically train at least a portion of the one or more artificial intelligence techniques using at least a portion of the generated speech data.

13. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to: implement one or more artificial intelligence techniques in association with one or more speech synthesis tasks; generate, in a plurality of successive portions, at least one data sequence comprising one or more of phonetic data and prosodic data by processing at least one previously generated data sequence using the one or more artificial intelligence techniques; and generate speech data corresponding to at least a portion of the data sequence by processing the at least a portion of the data sequence using at least one artificial intelligence-based speech synthesis model.

14. The computer program product of claim 13, wherein implementing one or more artificial intelligence techniques comprises: combine at least one LM, at least one TTS-FE model, and at least one TTS-P model.

15. The computer program product of claim 13, wherein implementing one or more artificial intelligence techniques comprises: convert at least a portion of output from at least one LM to phonetic data and prosodic data to be used as input to one or more of at least one TTS-FE model and at least one TTS-P model using a language model text-to-speech algorithm adapter.

16. The computer program product of claim 15, wherein implementing one or more artificial intelligence techniques comprises: use a language model text-to-speech algorithm adapter in conjunction with generating one or more HPC features derived from one or more statistical measures taken over one or more hierarchical time intervals and normalized to represent one or more speaker-independent features, wherein at least a portion of the one or more HPC features include at least one of one or more global assessments and one or more local assessments of at least one of a phonetic rate, a pitch, and a volume.

17. A computer-implemented method comprising: implementing one or more artificial intelligence techniques in association with one or more speech synthesis tasks; generating, in a plurality of successive portions, at least one data sequence comprising one or more of phonetic data and prosodic data by processing at least one previously generated data sequence using the one or more artificial intelligence techniques; and generating speech data corresponding to at least a portion of the data sequence by processing the at least a portion of the data sequence using at least one artificial intelligence-based speech synthesis model; wherein the method is performed by at least one computing device.

18. The computer-implemented method of claim 17, wherein implementing one or more artificial intelligence techniques comprises: combining the at least one LM, the at least one TTS-FE model, and the at least one TTS-P model.

19. The computer-implemented method of claim 17, wherein implementing one or more artificial intelligence techniques comprises: using a language model text-to-speech algorithm adapter to convert at least a portion of the output from the at least one LM into phoneme data and prosody data to be used as input to one or more of the at least one TTS-FE model and the at least one TTS-P model.

20. The computer-implemented method of claim 19, wherein implementing one or more artificial intelligence techniques comprises: using a language model text-to-speech algorithm adapter in conjunction with generating one or more HPC features, the one or more HPC features being derived from one or more statistical measures taken over one or more hierarchical time intervals and normalized to represent one or more speaker-independent features, wherein at least a portion of the one or more HPC features include at least one of one or more global assessments and one or more local assessments of at least one of phoneme rate, pitch, and volume.

Images