Speech synthesiser and method for speech synthesis

EP4463848A1Pending Publication Date: 2024-11-20SIEMENS AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
EP2023716409
Authority / Receiving Office
EP · EP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-05-17
Filing Date
2023-03-23
Publication Date
2024-11-20

AI Technical Summary

Technical Problem

Existing speech synthesis systems struggle to generate natural and emotionally nuanced speech melodies, as they rely on mechanical emotion recognition and lack human feedback, resulting in unnatural and less engaging artificial voices.

Method used

A speech synthesizer equipped with a neural network and AI, utilizing a microphone module for recording and processing human interactions, which incorporates feedback loops from human listeners to enhance emotional modeling and generate more natural speech melodies through training data expanded by human assessments.

Benefits of technology

The proposed solution significantly improves the naturalness and emotional authenticity of synthetic speech, making it more engaging and lifelike, particularly in Western languages, by integrating human feedback to refine pitch contours and emotional expressions.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention relates to a speech synthesiser and to a method for speech synthesis. Using the speech synthesiser proposed here for the first time and / or the method for speech synthesis proposed here for the first time, which make use of AI with a generic algorithm and at least one feedback loop through exchange with human listeners, voices can be artificially created with appropriate emotional modelling.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Description

[0002] Speech synthesizers and methods for speech synthesis

[0003] The invention relates to a speech synthesizer and a method for speech synthesis.

[0004] There are an increasing number of applications where it is advantageous if, in addition to a text output of a solution proposed by the computer-aided system - see, for example, navigation systems - spoken output modules also reproduce the solution that the system has calculated.

[0005] Natural language human-computer interaction is becoming increasingly popular and, thanks to deep learning algorithms, is also technically feasible. However, for a natural exchange between humans and computers, emotional modeling of the voice pitch is also necessary. The input variables and models required for this represent a technical challenge because the understanding, and therefore the synthetic reproduction, of emotions conveyed through speech melodies is not yet very well developed. For example, a significant majority of an audience can recognize whether a purely acoustically reproduced speaker is smiling or not during his speech.

[0006] The computer-aided systems with spoken output use speech synthesis and speech synthesizers as well as methods for speech synthesis.

[0007] Speech synthesis is the artificial creation of the human speaking voice. Text-to-speech systems already exist that convert continuous text into an acoustic speech output. These systems generally use robot voices to speak the text. Various techniques can be used for speech synthesis, including signal generation and signal modeling. Signal modeling relies on voice recordings, whereby the problem of generating a natural speech melody and / or emotional modeling is a particular challenge for any form of speech synthesis.

[0008] The object of the present invention is to create a speech synthesizer which improves existing systems with regard to speech melody and / or emotional modeling of artificial speech.

[0009] This object is achieved by the subject matter of the present invention as defined in the description and the claims.

[0010] Accordingly, the subject of the present invention is a speech synthesizer, the following modules, at least one processor and at least one neural network with an AI on which a generic algorithm is programmed, comprising:

[0011] - At least one microphone module with recording function

[0012] - At least one memory module that stores a recording of natural and / or artificially spoken language in the form of acoustic data and transmits it via a suitable interface to at least one

[0013] - processor which receives, analyses and processes the acoustic data from the memory module, wherein the processor is configured to have at least one speech analysis module which analyses and processes natural speech so that the correct statement is formulated,

[0014] - wherein the at least one processor is configured to also have an emotional module which carries out the emotional modelling of the statement in artificial language,

[0015] - wherein the two modules are connected to a neural network having an artificial intelligence (AI) that provides a suggestion regarding emotional modeling with regard to the content of the statement, wherein the AI ​​develops the suggestion for emotional modeling based on appropriate training data, which is at least partly generated by human interaction, and a generic algorithm, and

[0016] - Finally, a loudspeaker module for reproducing the synthetic speech.

[0017] The general finding of the invention is that, through generic algorithms, programming and corresponding feedback loops, including human interaction, AI solutions for emotional modeling of a speech can be provided in a speech synthesizer equipped in this way, which enable the generation of a very natural-sounding speech melody.

[0018] When operating the speech synthesizer, a learning process must first be carried out so that the AI's training data is supplemented by feedback from human listeners.

[0019] In contrast to known Speech Emotion Recognition (SER) systems, which recognize and map emotions automatically using frequencies, volume, rhythm, etc. via convolutional neural networks, the speech synthesizer presented here for the first time uses feedback loops with human listeners, so that the training data for the AI ​​used actually not only includes machine-capturable and technically measurable data, but is also based in particular on feedback from human listeners. This makes a significant difference because prosody, through human classification, allows for much more differentiated and far-reaching statements than the technical-machine recording used previously by well-known programs such as SER.

[0020] The feedback loops used by the speech synthesizer in learning mode are therefore based on the feedback of already known data available in libraries and / or programs of various programming languages, but also at least on the assessments of human listeners. For example, it has been found that human listeners can classify—with limitations, of course—but significantly accurately, whether a speaker is smiling or not while speaking in a purely acoustic playback. This human ability is captured by the speech synthesizer proposed here and converted into digitally processable data that can be used for speech synthesis via AI.

[0021] A prototype of the speech synthesizer resulted in a significant increase in the naturalness of the synthetic speech generated. This is primarily due to the fact that the pitch contour, i.e., the progression of the pitch, is trained with human interaction in the AI, resulting in a more lively and emotional appearance of the artificial language, especially in Western languages, rather than in Asian languages.

[0022] A computer architecture used in natural language processing and / or generation (NLP) is a neural network based on a deep learning model.

[0023] Common libraries and / or programs that also form the basis of the speech synthesizer described here can, for example, include language processing models, in particular autoregressive language processing models, such as GPT “Generative Pre-trained Transformer”. This refers to a series of Natural Language Processing (NLP) models, GPTI to GPT 3, in which deep learning is used to generate and / or process natural language.

[0024] Common libraries that can be used, for example, in the configuration of at least one processor and / or the programming of the AI ​​include Python, LIWC "Linguistic Inquiry Word Count", which are already used for computational linguistics applications. Emotional attributions via generic AI algorithms have so far also been carried out with the help of BERT, GPT3, GitHub and / or Copilot and are supplemented by the speech synthesizer proposed here.

[0025] By supplementing the training data for speech synthesis with feedback and feedback loops with human listeners, the pool of training data for the AI ​​described here can be expanded as desired, since queries that cannot be captured by machines, such as "how likely is it that the candidate passed the oral exam?" or "was the candidate hired?", which can be answered by the human listener not only based on content but also solely on the melody of the speech, can be incorporated into the AI's training. This speech synthesizer is therefore equipped with one or more microphones.

[0026] The training data is generated by human listeners, particularly with regard to the emotions associated with the recording. For example, questions are asked as to whether the speaker is smiling, speaking authentically, whether one "believes" what they are saying, or whether uncertainties can be detected in the speech intonation. This can be done in connection with the content and / or independently of the content, for example by listening to an unfamiliar human speech. The training data can then be transferred as required to different types of content that, for example, do not match in any word or meaning, but require the human speaker to use the same speech intonation.

[0027] This is then the training of the AI ​​to recognize which content the speech melody fits.

[0028] For example, certain moods are also assigned to certain contents in relation to speech melodies: The following emotions can be assigned to the respective acoustic data with varying degrees of accuracy:

[0029] Does the speaker express admiration, pleasure, fear, anger, approval, compassion, confusion, curiosity, desire, disappointment, disapproval, disgust, embarrassment, excitement, anxiety, gratitude, sorrow, joy, love, nervousness, optimism, pride, or insight? If so, to what extent?

[0030] For example, the human listener can express his perception in numerical values ​​from 1 to 10.

[0031] So-called human emotion recording modules are provided to capture these human emotions and convert them into machine-processable data. For example, a human listener can operate various input modules while listening. This way, the human can sit in front of a series of controls and sliders, each representing a different emotion. This way, the human can continuously move the control "gratitude, disapproval, anger, fear, irony, pleasure..." etc. while listening, thereby supplying the data the AI ​​needs to train the generic process. The same inputs can also be made by the human via touchscreen and / or keyboard.

[0032] For training purposes, human speakers are recorded in various situations and analyzed accordingly, whereby a value can be assigned to each mood so that the training data for a speech melody can be assigned to as many moods as possible in connection with the analyzed content of the statement.

[0033] From this training data, which is generated with the help of the microphones and the simultaneous classifications by human listeners, the AI ​​can suggest speech melodies for specific content in connection with a content, which are then reproduced very authentically by the loudspeaker.

[0034] The AI's training data is generated on the one hand by feedback loops of human speech with human listeners and on the other hand by feedback loops of synthetically generated speech with human listeners.

[0035] It is advantageous if the microphone(s) forming part of the speech synthesizer have one or more filters that precisely capture the pitch of the spoken speech, whether synthetic or human. Suitable filters for sound recordings can meet various requirements regarding noise rejection and are known to those skilled in the art.

[0036] It is also advantageous if the microphone(s) have devices that also record breathing sounds. For example, the microphone may comprise two or more channels through which the speech and breathing sounds are recorded simultaneously, but on different audio tracks, so that the two recordings can be listened to separately or combined in feedback loops.

[0037] Furthermore, it is advantageous if the microphone(s) recognize(s) the exact position of the speaker, so that a confident speech directed directly at the microphone provides different acoustic data for processing by the AI ​​than a withdrawn speech with the head perhaps even lowered and / or turned away from the microphone, with otherwise the same pitch etc., i.e. pitch contour.

[0038] The storage module for storing the generated acoustic data is advantageously configured to convert the acoustic data into machine-readable data. For example, the storage module can have a program for compressing the data.

[0039] According to an advantageous embodiment, the storage module is designed to allow a comparison of the data with previously stored acoustic data, so that data that is repetitive and therefore does not add value to the AI ​​training is at least not forwarded, but in particular is separated, for example. This data can then be stored elsewhere or even deleted.

[0040] In the context of the invention, a “processor” can be understood as meaning, for example, a machine and / or an electronic circuit. A processor can in particular be a main processor (Central Processing Unit, CPU), a microprocessor or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc. A processor can also be, for example, an IC (Integrated Circuit), in particular an FPGA (Field Programmable Gate Array) or an AS IC (Application-Specific Integrated Circuit), or a DSP (Digital Signal Processor) or a graphics processor GPU (Graphic Processing Unit).

[0041] A processor can also be understood as a virtualized processor, a virtual machine, or a soft CPU. It can also be, for example, a programmable processor that is equipped with configuration steps for executing the aforementioned method according to the invention or is configured with configuration steps such that the programmable processor implements the inventive features of the method or modules, or other aspects and / or sub-aspects of the invention.

[0042] In the context of the invention, a "module" can be understood as, for example, a device such as a microphone, a camera and / or a storage unit for storing acoustic and / or visual data. For example, the processor is specifically designed to execute the digital representation in such a way that the AI ​​carries out functions in order to implement or realize pattern analysis, pattern recognition and / or pattern prediction and / or a step of the method according to the invention. The respective modules can, for example, also be designed as separate or independent modules. For this purpose, the corresponding modules can, for example, comprise further elements. These elements are, for example, one or more interfaces (e.g. database interfaces, communication interfaces - e.g. network interface, WLAN interface) and / or an evaluation unit (e.g. a further processor) and / or a storage unit.Using the interfaces, for example, data can be exchanged (e.g., received, transmitted, sent, or provided). Using the evaluation unit, data can be compared, checked, processed, assigned, or calculated, for example, in a computer-aided and / or automated manner. Using the storage unit, data can be stored, retrieved, or provided, for example, in a computer-aided and / or automated manner.

[0043] The invention also relates to a method for speech synthesis, comprising the following method steps: a) playing back a synthetic and / or human speech, b) simultaneously recording one or more human reactions to this speech in real time, c) converting the recorded data into machine-processable data, d) saving the data, e) repeating steps a) to d), f) forwarding this data as training data to a neural network which is configured in such a way that it supplies solutions for speech synthesis using generic programming and taking this data into account, g) implementing the suggestions for speech synthesis generated by the AI ​​using a suitably configured processor, and h) outputting the synthesized speech.

[0044] The repetition cycles of process steps a) to d) are arbitrary and can be between 1 and 10,000, in particular between 1 and 1,000 or between 1 and 100 repetitions. "Prosody" is the totality of those phonetic properties of language that are not tied to the sound and / or the phoneme as a minimal segment, but to more comprehensive phonetic units. These include the following properties: word and sentence accent; the lexical tone resting on word syllables in tonal languages. Prosody comprises the typical speech rhythm, intonation and / or stress of a language.

[0045] This speech synthesizer and / or this speech synthesis method generates an artificial language taking prosody into account.

[0046] In this way, any technically unmeasurable data in connection with a speech can be used to train the AI, for example: speech rhythm, speech melody in connection with content, which in turn allows conclusions to be drawn about eye contact, facial expressions, head posture and, above all, data that cannot be recorded at all from a sensory perspective, such as

[0047] "did the human / synthetic speech seem authentic?" or

[0048] "Did human / synthetic speech seem awkward?" "Stressed?" "Focused?" "Calm?" "Friendly?" "Did his mood change during the speech?" "Did he maintain a constant speaking rate?"

[0049] These conclusions should also be able to be drawn from the synthetic speech so that the synthetic speech appears as natural as possible.

[0050] The proposed speech synthesizer and / or the proposed speech synthesis method do not require an explicit understanding of the modeling of emotions in the voice pitch; instead, they rely on feedback from human interaction. The only important thing is that it works, that the emotion is recognizable in the artificial voice, not the understanding of how this happens.

[0051] In addition, indirect emotions can be captured via the acoustic data, such as "did the human speaker use many or few filler words?" "did this change during the speech?" "when did filler words appear?" for certain content such as "uh, hmm, clearing throat..." When were there pauses in speech? How did the speech melody relate to the content? The learning process must be distinguished from the application.

[0052] AI is based on the well-known speech synthesis tools and extends these tools with data generated by human interaction.

[0053] It is advantageous if a classification of the various learned emotional colorings is provided so that a quick definition by the user is possible.

[0054] By means of the speech synthesizer and / or the method for speech synthesis proposed here for the first time, which uses an AI with a generic algorithm and at least one feedback loop through the exchange with human listeners, voices with corresponding emotional modeling can be artificially created.

Claims

Patent claims 1. Speech synthesizer comprising the following modules, at least one processor and at least one neural network with an AI on which a generic algorithm is programmed: - At least one microphone module with recording function - At least one storage module that stores a recording of naturally and / or artificially spoken speech in the form of acoustic data and connects it via a suitable interface to at least one - Processor forwards the acoustic data from the memory module, analyzes and processes it, wherein the processor is configured to have at least one speech analysis module that analyzes and processes natural language so that the correct statement is formulated in terms of content, - where at least one processor is configured to also have an emotional module that handles the emotional modeling of the statement in artificial language, - wherein the two modules are connected by a neural network that has an artificial intelligence (AI) which provides a proposal regarding the emotional modeling with respect to the content of the statement, wherein the AI ​​develops the proposal for emotional modeling based on appropriate training data, which is generated at least partly through human interaction, and a generic algorithm and - Finally, a speaker module for playing back the synthetic speech.

2. Speech synthesizer according to claim 1, comprising a speech processing model that utilizes a deep learning architecture to generate human-like text.

3. Speech synthesizer according to one of claims 1 or 2, comprising an interface to a library.

4. Speech synthesizer according to one of the preceding claims, comprising a module for capturing human emotions and a series of controls, each of which can be assigned to other emotions.

5. Speech synthesizer according to one of the preceding claims, comprising at least one microphone module having at least one filter for noise selection.

6. Speech synthesizer according to one of the preceding claims, comprising at least one microphone module that records not only speech but also breathing sounds.

7. Speech synthesizer according to one of the preceding claims, comprising at least one memory module suitable for comparing recorded data with existing data.

8. Speech synthesizer according to any of the preceding claims, comprising at least one memory module suitable for compressing incoming data.

9. Speech synthesis method comprising the following steps: i) playing back synthetic and / or human speech, j) capturing one or more human responses to this speech in real time, k) converting the captured data into machine-processable data, l) storing the data, m) repeating steps a) to d), n) forwarding this data as training data to a neural network configured to provide speech synthesis solutions via generic programming and taking this data into account. o) Implementing the AI-generated speech synthesis suggestions using a suitably configured processor and p) Outputting the synthesized speech.

10. Method according to claim 9, wherein the human response with regard to emotions such as: admiration, pleasure, fear, anger, agreement, compassion, confusion, curiosity, desire, disappointment, disapproval, disgust, embarrassment, excitement, anxiety, gratitude, sorrow, joy, love, nervousness, optimism, pride, recognition is recorded during the speech.

11. The method of claim 10, wherein the extent of the human reaction with regard to emotions is recorded.

12. Method according to one of claims 9 to 11, wherein filler words in speech are captured.

13. Method according to any one of claims 9 to 12, wherein the speaker's breathing sounds are recorded.

14. Method according to any one of claims 9 to 13, wherein the method steps a) to d) are repeated a maximum of 1000 times.

15. Method according to any one of claims 1 to 14, wherein a classification of the different learned emotional models is provided.