SPEAKER-SPECIFIC ADAPTING OF A MACHINE-LEARNED SPEECH SYNTHESIS MODEL FOR VOICELESS SPEAKERS

Abstract

Techniques are described that enable the generation of speaker-specific training samples in a training dataset for training a machine-learned speech synthesis model, even for voiceless speakers. This allows for speaker-specific training of the machine-learned speech synthesis model. The machine-learned speech synthesis model can be regularly retrained, for example, if the anatomy of a particular speaker and / or the hardware used in the measurement modality changes. This enables speech synthesis to be provided with increased accuracy.

Description

TECHNICAL AREA

[0001] Several examples of revelation described herein relate generally to the speaker-specific adaptation of a machine-learned speech synthesis model. Several examples described herein specifically enable the speaker-specific adaptation of a machine-learned speech synthesis model for voiceless speakers. BACKGROUND

[0002] Using articulatory measurement modalities – e.g., ultrasound, electromyography, or radar – it is possible to recognize features of silent speech movements (for example, for people with complete or partial voice loss (voiceless speakers) or even speakers who voluntarily speak silently) using a machine-learned speech synthesis model and to convert them into acoustic speech signals. Corresponding techniques are known, for example, from: DE 10 2020 110 901 A1.

[0003] The machine-learned speech synthesis model is trained using training samples in a training dataset. These training samples contain a measurement data stream, captured using the respective articulatory measurement modality, which serves as input for the machine-learned speech synthesis model. As the basis for the machine-learned model's output, the training samples include a corresponding speech utterance associated with the respective articulation. The machine-learned speech synthesis model can then be trained to translate measurement data streams into corresponding synthetic speech utterances.

[0004] Traditionally, the training samples in the training dataset include audio files containing speech utterances from healthy reference speakers.

[0005] For these healthy speakers, who possess articulatory ability, corresponding measurement data streams are recorded along with the acoustic recording of the speech utterance as an audio file for each training sample. However, due to anatomical and technical variations, it cannot necessarily be expected that a machine-learned speech synthesis model, trained on samples for one or more healthy reference speakers, will exhibit good accuracy for other speakers (so-called out-of-distribution speakers). To enable generalization to new speakers, additional training samples can be recorded for the new speaker. Based on these additional training samples, the pre-trained machine-learned speech synthesis model can then be further refined (fine-tuned). This is also referred to as speaker-specific adaptation.

[0006] The following describes various techniques for speaker-specific adaptation of a machine-learned speech synthesis model. The fundamental problem for voiceless speakers is that no acoustic speech utterance can be recorded for them. Therefore, existing technologies offer approaches to achieve speaker-specific adaptation in other ways. Such examples are highlighted below.

[0007] In some well-known reference implementations, domain adaptation is achieved by transforming the input characteristics in a captured measurement data stream (hereinafter referred to as X). artThese include Feature Space Maximum Likelihood Linear Regression (fMLLR) and Procrustes Analysis, which were used for silent speech recognition with surface myography (sEMG) [1] and electromagnetic articulography (EMA) [2]. In fMLLR, a global linear mapping between the measurement data stream and the model input is learned from the training data; the machine-learned speech synthesis model (hereinafter referred to simply as the model) itself remains unchanged. This mapping is refined for new speakers with a new set of training data to compensate for domain shift. Procrustes Analysis was used in speech recognition with EMA to normalize the measured trajectories of the individual coils in the midsagittal plane with respect to their geometric variance (rotation, scaling, and translation) [2].This group also includes all reference implementations that aim for pure input feature normalization.

[0008] In some reference implementations, domain adaptation is achieved by adjusting the machine-learned speech synthesis model. Wand et al. used so-called domain adversarial training [3] and meta-learning [4] for silent speech recognition with the aim of domain adaptation to a new recording session rather than to completely new speakers, with both reference implementations also being applicable for adaptation to new speakers. The idea behind domain adversarial training is the simultaneous optimization of the model parameters with two competing loss functions, similar to generative adversarial networks

[14] . The model consists of a feature extractor whose output is the input for two further, parallel models. Model 1 is trained "classically" with a loss function corresponding to the problem, e.g.The mean squared error for the regression problem of mapping continuous articulatory input features to equally continuous output features for speech synthesis. Model 2 is a classifier with the number of available sessions or speakers in the training dataset as possible classes, whose gradients are negated during training, thereby implicitly maximizing loss function 2. This prevents the model from learning domain-invariant features and, ideally, allows it to generalize to new sessions or speakers without the need for fine-tuning. If the domain shift is too large, fine-tuning can be performed using only measurement data streams, without parallel audio data. Meta-learning, on the other hand, aims to reduce the amount of training data required for subsequent domain adaptation by simply adjusting the training strategy while using the same training dataset [5].None of the methods mentioned so far explicitly solve the problem of the missing audio signal from a voiceless speaker, which is addressed by Gatty et al. [6]. The basic idea is to have the same silently articulated sentence also vocalized by auxiliary speakers, one of whom X. art,vokalisiert and X aku (acoustic speech utterance, i.e., audio file) is recorded. During the training process, pairs of {X art,still , X art,vokalisiert , X aku} created with the same sentence content and aligned against each other using the Dynamic Time Warping (DTW) algorithm

[17] . The missing audio signal of the voiceless speaker X aku,still This corresponds to a time-adjusted version of X. aku , which is a direct mapping of X art,still on the aligned row X aku,stillThis method has been improved several times with additional regularization methods for pathfinding [8,12] and has also been investigated for EMA

[11] .

[0009] In the field of silent speech synthesis using ultrasound, spatial transformer networks

[16] have been applied to achieve higher geometric invariance of the input features (in this case, images of the base of the tongue in the medio-sagittal plane)

[13] , similar to Procrustes analysis. However, true domain adaptation does not occur here, as the model can only learn speaker invariance for the speakers present in the training dataset, which does not necessarily generalize to unseen speakers. The same authors proposed the use of so-called neural speaker embeddings or X-vectors

[18] based on X. artThis method is similar to those already used in acoustic speaker identification. For each speaker, an X-vector is calculated from an articulatory measurement series of a short sentence, analogous to the calculation from the audio signal itself. During training, the model is conditioned with these X-vectors. For an unseen speaker, an X-vector is initially calculated, which then serves as a permanent, additional model input during synthesis.

[0010] Cao et al. [9,10] used a text-to-speech (TTS) model. In this reference implementation, the articulatory EMA measurement data were mapped to phonemes, which were then synthesized by a pre-trained TTS model. Domain adaptation was achieved through simple fine-tuning with newly recorded data for each speaker. Using a phoneme sequence as the target variable circumvented the problem of missing audio signals. The authors also experimented with various augmentation methods to regularize and reduce domain shift for a new speaker.

[0011] The following table summarizes the disadvantages of the existing solutions. TABLE 1. Disadvantages of various previously known techniques for speaker adaptation. method Domain adaptation mechanism Fine-tuning is necessary (i.e., always retraining with new, speaker-specific data). Disadvantage fMLLR Adaptation of the global-affine mapping of the input characteristics Yes, the affine mapping with {X art ,X aku }-pairs Empirically proven only for small domain shifts. Does not solve the problem of missing X. aku Procrustes Analysis & Normalization Speaker-independent input features help with the generalization of the model. Either with X art or no need Empirically proven only for small domain shifts. Does not solve the problem of missing X. aku Domain adversary I training Ideally, the model learns a domain-independent representation of the input features. Yes, either with {X art , X aku }-pairs or just X art There is no guarantee that the model will learn domain-independent representations and therefore does not completely solve the problem of missing X. aku Meta-learning The model learns a better internal representation, which enables faster adaptation. Yes, with {X art , X aku }-Pair -"- DTW matching DTW matching provides the missing, alienated audio signal to the articulatory signal. Yes, with {X art , X aku }-Pair X art The auxiliary speaker is necessary, and therefore the amount of fine-tuning data is limited. STN Normalization of the input features and / or intermediate representations No Large domain shifts still require fine-tuning. Only applicable to image-processing measurement methods. X-vectors The model is conditioned with X-vectors to better generalize to unseen speakers. Yes, by recording a single X art and calculation of the X-vector No successful adaptation has been demonstrated so far. X art -to-phoneme with TTS synthesis Mapping to phoneme classes is potentially simpler than mapping to continuous features (e.g., mel spectra) because it is free from speaker-specific factors. Yes, with {X art ,phoneme sequence} pairs X art Phoneme-to-phoneme transcription has a latency of at least one phoneme length. Phoneme boundaries are sometimes fluid. This depends on the quality of phoneme segmentation and alignment. Features that must be learned (or learned) Algorithm and static phoneme vocabulary Sources: [1] Wand, Michael, and Tanja Schultz. „Speaker-Adaptive Speech Recognition Based On Surface Electromyography.“ International Joint Conference on Biomedical Engineering Systems and Technologies, Berlin, Heidelberg, 2009 [2] Wang, Jun, Ashok Samal, and Jordan Green. „Across-Speaker Articulatory Normalization For Speaker-Independent Silent Speech Recognition.“ Interspeech, Singapore, 2014 [3] Wang, Jun, and Seongjun Hahm. „Speaker-Independent Silent Speech Recognition With Across-Speaker Articulatory Normalization And Speaker Adaptive Training.“ Interspeech, Dresden, Germany, 2015 [4] Wand, Michael, Tanja Schultz, and Jürgen Schmidhuber. „Domain-Adversarial Training For Session Independent EMG-Based Speech Recognition.“ Interspeech, Hyderabad, India, 2018 [5] Proroković, Krsto, et al. „Adaptation Of An EMG-Based Speech Recognizer Via Meta-Learning.“ IEEE Global Conference on Signal and Information Processing (GlobalSIP), Ottawa, Ontario, Canada, 2019 [6] Gaddy, David, and Dan Klein. „Digital Voicing Of Silent Speech.“ arXiv preprint arXiv:2010.02960, 2020 [7] Shandiz, Amin Honarmandi, et al. „Neural Speaker Embeddings For Ultrasound-Based Silent Speech Interfaces.“ arXiv preprint arXiv:2106.04552, 2021 [8] Gaddy, David, and Dan Klein. „An Improved Model For Voicing Silent Speech.“ arXiv preprint arXiv:2106.01933, 2021 [9] Cao, Beiming, et al. „Investigating Speech Reconstruction For Laryngectomees For Silent Speech Interfaces.“ Interspeech, Brno, Czechia, 2021

[10] Cao, Beiming, et al. „Data Augmentation For End-To-End Silent Speech Recognition For Laryngectomees.“ Interspeech, Incheon, Korea, 2022

[11] Cao, Beiming, Alan Wisler, and Jun Wang. „Speaker Adaptation On Articulation And Acoustics For Articulation-To-Speech Synthesis.“ Sensors, 6056, 2022

[12] H. Li et al., „Silent Speech Interface With Vocal Speaker Assistance Based On Convolution-Augmented Transformer,“ in IEEE Transactions on Instrumentation and Measurement, vol. 72, pp. 1-11, 2023

[13] Tóth, László, et al. „Adaptation Of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks.“ arXiv preprint arXiv:2305.19130, 2023

[14] Goodfellow, lan, et al. „Generative Adversarial Nets.“ Proc. Of the 27th Advances in Neural Information Processing Systems, Montreal, Canada, 2014

[15] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. „Model-Agnostic Meta-Learning For Fast Adaptation Of Deep Networks.“ International Conference on Machine Learning. PMLR, Sydney, Australia, 2017

[16] Jaderberg, Max, Karen Simonyan, and Andrew Zisserman. „Spatial Transformer Networks.“ Proc. Of the 28th Advances in Neural Information Processing Systems, Montreal, Canada, 2015

[17] H. Sakoe and S. Chiba, „Dynamic Programming Algorithm Optimization For Spoken Word Recognition,“ in IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 26, no. 1, pp. 43-49, February 1978

[18] Snyder, David, et al. „X-Vectors: Robust DNN Embeddings For Speaker Recognition.“ International Conference On Acoustics, Speech and Signal Processing (ICASSP), Calgary, Alberta, Canada, 2018 KURZE BESCHREIBUNG

[0012] There is therefore a need for improved techniques for speaker-specific adaptation of machine-learned speech synthesis models that generate speech utterances based on articulatory measurement modalities. In particular, there is a need for techniques that enable speaker-specific adaptation even for voiceless speakers.

[0013] This task is solved by the features of the independent patent claims. The features of the dependent patent claims define embodiments.

[0014] The following describes techniques that enable the generation of speaker-specific training samples in a training dataset for training a machine-learned speech synthesis model, even for voiceless speakers. This allows for speaker-specific training of the machine-learned speech synthesis model. The machine-learned speech synthesis model can be regularly retrained, for example, if the anatomy of a particular speaker and / or the hardware used in the measurement modality changes. This enables speech synthesis to be provided with increased accuracy.

[0015] A computer-implemented method for speaker-specific adaptation of a machine-learned speech synthesis model is disclosed. The machine-learned speech synthesis model translates measurement data streams from an articulatory measurement modality into synthetic speech utterances. The method comprises populating a training dataset with speaker-specific training samples and initiating the training of the machine-learned speech synthesis model based on the training dataset. Populating the training dataset is performed iteratively.Each of these iterations includes generating a corresponding audio file using a text-to-speech model for a synthetic utterance that corresponds to a specific text body; driving a loudspeaker to play back the synthetic utterance based on the corresponding audio file; and capturing a corresponding measurement data stream using the articulatory measurement modality for the speaker's articulation associated with the synthetic utterance. Furthermore, each iteration optionally includes incorporating the corresponding measurement data stream into the training dataset.

[0016] The method can further include inferring the machine-learned speech synthesis model based on additional measurement data streams. No fundamental truth exists for such additional measurement data streams. By inferring the machine-learned speech synthesis model, synthetic speech utterances are obtained for these additional measurement data streams.

[0017] An electronic data processing device is configured to perform such a procedure. For this purpose, the electronic data processing device includes a processor and memory. The processor can load and execute program code from memory. When the processor executes the program code, it performs the steps of such a procedure.

[0018] The features set out above and those described below can be used not only in the corresponding explicitly set out combinations, but also in further combinations or in isolation, without leaving the scope of protection of the present invention. BRIEF DESCRIPTION OF THE FIGURES Fig. Figure 1 is a flowchart of an example procedure. Fig. Figure 2 is a flowchart of an example procedure. Fig. Figure 3 shows a system with a sensor device for an articulatory measurement modality and an electronic data processing device according to various examples. DETAILED DESCRIPTION

[0019] The properties, features and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more easily understood in connection with the following description of the exemplary embodiments, which are explained in more detail in conjunction with the drawings.

[0020] The present invention is explained in more detail below with reference to preferred embodiments and the drawings. In the figures, identical reference numerals denote identical or similar elements. The figures are schematic representations of various embodiments of the invention. Elements depicted in the figures are not necessarily shown to scale. Rather, the various elements depicted in the figures are represented in such a way that their function and general purpose are understandable to a person skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as indirect connections or couplings. A connection or coupling can be implemented as a wired or wireless connection. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0021] The following describes techniques for speaker-specific adaptation of a machine-learned speech synthesis model. The machine-learned speech synthesis model uses input data based on one or more measurement data streams. These one or more measurement data streams are acquired using one or more articulatory measurement modalities.

[0022] Examples of articulatory measurement modalities include: optical lip reading, ultrasound imaging, electromyography (EMG), electroencephalography (EEG), electropalatography (EPG), electromagnetic articulography (EMA), permanent magnet articulography (PMA), strain gauges; and high-frequency measurements, for example, in the ultra-wideband (UWB) range, especially frequency-modulated continuous wave radar (FMCW) transmitting through the oral cavity. In general, the various techniques can be particularly effective when different properties of the speech apparatus, especially the vocal tract, are comprehensively measured. Therefore, in several of the techniques described herein, it can be advantageous to use articulatory measurement modalities that can specifically measure properties of the oral cavity and pharynx of the vocal tract. Examples of such articulatory measurement modalities are radar measurements and ultrasound measurements.These measurements can be made using extracorporeal sensors, which are attached to the skin surface, for example, with a medical patch. Simultaneously, it may be desirable to combine such articulatory measurement modalities, which measure properties within the vocal tract, such as in the oral cavity, with other articulatory measurement modalities that measure lip and / or tongue movement. These could be camera-based techniques. In this way, the vocal tract can be comprehensively characterized. In particular, it has been shown that radar measurements make it possible to comprehensively measure large areas of the vocal tract. For example, jaw opening, tongue positioning, and lip movement can be measured.

[0023] Generally, the speaker-specific adaptation of a machine-learned speech synthesis model is referred to as domain adaptation. The goal of domain adaptation is to reduce or eliminate the so-called domain shift. Domain adaptation of models specifically for silent speech synthesis from articulatory data represents a special case in the research field of machine speech processing and is based on methods developed for acoustic speech synthesis and recognition or transcription. A domain shift occurs primarily due to (a) a change of speaker, (b) disconnecting and reconnecting the measuring device (so-called session change), or (c) changes in the measurement signal resulting from hardware modifications. In principle, it is conceivable to implement the speaker-specific adaptation techniques of the machine-learned speech synthesis model described here for each of the cases (a), (b), or (c).

[0024] In principle, different types of machine-learned speech synthesis models can be used for the techniques described herein. In particular, machine-learned speech synthesis models can be used that have a corresponding coding branch for each measurement modality (i.e., for each measurement data stream). The corresponding latent feature vectors can then be combined and decoded. If speaker-dependent speech synthesis is desired, this can be achieved during decoding (i.e., mapping the articulatory feature space to the audio space); for example, vocoders are known that take speaker-specific characteristics (e.g., voice quality or timbre, fundamental frequency, intonation, prosody, articulation speed, nasality, etc.) into account. Examples include WaveNet and Tacotron, which can be individually adapted to speakers.It would be conceivable that only one or more coding branches are adjusted during refinement training. The vocoder can remain unchanged.

[0025] The refinement of a machine-learned speech synthesis model to adapt to a specific speaker is based on a training dataset containing a variety of speaker-specific training samples. Each training sample has a measurement data stream for a corresponding articulatory measurement modality and an associated audio file that reproduces a speech utterance for that speaker. The measurement data stream observes a specific articulation of the speaker, and the synthetic speech utterance is determined for that articulation.

[0026] For a voiceless speaker – voicelessness referring to the loss of phonation ability – speech cannot be directly recorded using an audio file due to the lack of a voice. The following describes various techniques that nevertheless enable speaker-specific training.

[0027] In the various examples, an audio file is generated as a reference for the speaker: In the various examples described herein, a TTS model is used to convert a specific text corpus into a corresponding TTS audio file containing the synthetic speech utterance associated with that text corpus. This audio file, generated using a TTS model, is then played back as a reference: The voiceless speaker is prompted to listen to the synthetic speech utterance generated in this way and to articulate it in the same manner. A measurement data stream is recorded using the articulatory measurement modality for this articulation of the speaker, that is, for the speaker's articulation associated with the synthetic speech utterance from the TTS model. This measurement data stream can then be included in the training dataset, for example, if one or more criteria are met.

[0028] Using the TTS audio file as a reference ensures that the speaker uses a specific articulation for which certain prior knowledge exists. This makes it easier to generate a suitable audio file – even for a voiceless speaker – and include it, along with the corresponding measurement data stream, as a training sample in the training dataset.

[0029] In one variant, the measurement data stream is included in the training dataset along with the TTS audio file as a training sample. However, a different audio file could also be used, for example, one that reproduces an artificial speech utterance of the speaker generated using a (conventional) substitute voice.

[0030] Including the TTS audio file in the training dataset along with the measurement data stream offers the advantage of allowing the TTS audio file to be specifically adapted to the individual speaker by appropriately adjusting the TTS model. Such speaker-specific adaptation can be achieved inherently through the use of a substitute voice. Conversely, using a separate substitute voice can be complex or even necessitate the existence of such a voice.

[0031] To decide whether a particular measurement data stream is included in the training dataset, it can be checked, for example, whether the measurement data stream and the synthetic speech utterance obtained by the TTS model are synchronized. More generally, a metric can be determined that is indicative of the synchronicity of the speaker's corresponding articulation (captured by the measurement data stream) with the corresponding synthetic speech utterance (reproduced in the TTS audio file). If, for example, the duration of the TTS audio file or the synthetic speech utterance deviates too much from the duration of the measurement data stream for the corresponding articulation, the process described above can be repeated in a further iteration for the same text. In particular, the synthetic speech utterance or theThe TTS audio file can be regenerated with an adjusted speaking rate, for example, by re-inferring the TTS model. If, on the other hand, the synchronicity metric indicates sufficient synchronicity, the corresponding measurement data stream can be included in the training dataset as an additional speaker-specific training sample.

[0032] Using the techniques described above, the training dataset can be populated with speaker-specific training samples by repeating the process for different text bodies—each associated with a different training sample. This results in a training dataset containing (quasi-)synchronous sensor-audio data pairs for a voiceless speaker. The techniques described above enable speaker-specific adaptation of a machine-learned speech synthesis model, even for voiceless speakers. Furthermore, the described method, which is based on "repeating" a synthetic speech utterance, is also accessible to illiterate individuals.

[0033] Fig. Figure 1 is a flowchart of an example procedure. The procedure is from Fig.1. This serves to adapt a machine-learned speech synthesis model to the speaker. The machine-learned speech synthesis model translates a measurement data stream obtained via an articulatory measurement modality into a synthetic speech utterance. The procedure from Fig. 1 is performed by an electronic data processing device. For example, a processor can load and execute program code from memory. When the processor loads and executes the program code from memory, this causes the processor to perform the procedure from Fig. 1 executes.

[0034] Box 3005 contains a training dataset with speaker-specific training samples. Each training sample includes a corresponding pair of a measurement data stream and an associated audio file. The measurement data stream is captured using an articulatory measurement modality for a specific articulation of the speaker. This articulation follows a reference specification. The reference specification is provided by a TTS audio file, which is obtained using a TTS model for a specific text. The speaker listens to the TTS audio file and articulates according to the synthetic speech utterance.

[0035] In Box 3010, training of the machine-learned speech synthesis model is initiated based on the speaker-specific training samples in the training dataset from Box 3005. In particular, refinement training can be performed; in other words, the machine-learned speech synthesis model may already be pre-trained based on other training datasets (for example, for generic speakers), and the weights of the machine-learned speech synthesis model are then further adjusted based on the speaker-specific training samples, typically starting from their previous values.

[0036] Training of the machine-learned speech synthesis model can generally be performed on a cloud server; in such a case, an electronic data processing device can load the training dataset into Box 3005 and then send the training dataset to the cloud server via a suitable communication interface (e.g., via the internet). The electronic data processing device can then receive the trained machine-learned model from the cloud server.

[0037] Once the training of the machine-learned speech synthesis model has been performed in Box 3010, Box 3015 can then be executed. In Box 3015, the machine-learned speech synthesis model is inferred to generate synthetic speech.

[0038] In Fig.Figure 1 shows that multiple iterations of Box 3005, Box 3010, and Box 3015 can be performed. In particular, it is possible to perform refinement training of the machine-learned speech synthesis model from time to time or when certain conditions occur. For example, refinement training could be performed when one or more sensors of the articulatory measurement modality—for example, an antenna for radar measurement—are newly attached to the speaker's skin.

[0039] Populating the training dataset in Box 3005 can be done iteratively. A corresponding example implementation of Box 3005 is shown in Fig. 2 illustrated.

[0040] Fig. Figure 2 is a flowchart of an example procedure. Optional blocks are shown in Fig. 2 shown with dashed lines. Fig.Section 2 comprises an outer loop with multiple iterations (3198) and an inner loop with multiple iterations (3199). The outer loop with multiple iterations (3198) defines several training samples, each assigned to a different text body (see Box 3105, where a corresponding text body is loaded from a corresponding text body database for each iteration (3198)). The inner loop with multiple iterations (3199) can be executed multiple times for each training sample or corresponding text body. Within the multiple iterations (3199), one or more parameter values ​​for one or more parameters associated with speech rate can be adjusted. The procedure is described below. Fig. 2 explained in detail.

[0041] First, in iteration 3198 of Box 3105, a corresponding text body is loaded. The text body can contain, for example, written text or a sequence of phonemes. The text body can be loaded into Box 3105 from a text body database.

[0042] There are different strategies for selecting the text body in Box 3105. For example, the selection could be random or at least partially random. This means that a specific text body could be randomly selected from the set of available text bodies for each iteration 3198. Alternatively, a systematic selection of the text body in Box 3105 could be made. For example, the selection could depend on the length of the respective text body. Shorter text bodies could tend to be selected in earlier iterations 3198 than in later iterations 3198. Typically, it is comparatively more difficult to articulate longer text bodies; by selecting shorter text bodies in earlier iterations 3198, the speaker can be introduced to more complex tasks. Another selection criterion would be the coverage of certain phonemes.For example, in a specific iteration 3198, a text body could be selected based on text bodies selected in previous iterations 3198. For instance, a particular text body could be selected in iteration 3198 such that it contains a specific sequence of phonemes that were not present, or only present in small numbers, in the text bodies from previous iterations 3198. In this way, the training dataset, populated with the corresponding training samples, can be made to contain a wide range of different phoneme sequences. This allows for better and more comprehensive speaker adaptation of the machine-learned speech synthesis model.

[0043] Then, in a corresponding first iteration 3199, Box 3110 is executed. In Box 3110, a TTS model is inferred in order to determine a synthetic speech utterance for the current text body from the current iteration 3198 of Box 3105.

[0044] There are various TTS models available in the state of the art. For example, machine-learned TTS models are known. TTS models that modify a measured speech utterance based on the text corpus could also be used. In principle, various TTS models available in the state of the art can be employed. In particular, it is helpful to use configurable TTS models that, based on corresponding control data, can adjust the speech rate by setting one or more relevant parameter values.

[0045] The inference of the TTS model can be performed, for example, using control data for the TTS model, where this control data specifies values ​​for at least one parameter associated with speech rate. Examples of such parameters include tempo, rhythm, or pauses. Some TTS models have a control interface to which such control data can be passed. By setting appropriate values, the global duration and / or the local time dynamics for specific passages of the text body are determined. When Box 3110 is executed, such control data can be predefined. For example, the user can specify the relevant control data beforehand via a corresponding interface. It would also be conceivable that such control data is randomly initialized or initialized with default values ​​in the first iteration 3198.

[0046] The TTS model can also be configured based on prosody control data associated with the speaker. This prosody control data can specify inaudible features of speech that are not tied to individual phonemes as minimal segments, but rather to broader sound units. This can include, for example, word or sentence accentuation, or the pace, rhythm, or pauses in speech. Intonation and sentence melody can also be adjusted. By incorporating such prosody control data, it becomes easier for the speaker to articulate the synthetic speech with appropriate temporal dynamics. This is because the synthetic speech already exhibits certain speech characteristics specific to the speaker.To ensure that the prosody control data matches the respective speaker, the prosody control data can be set by the user, for example, based on user input.

[0047] As a result of Box 3110, a TTS audio file for the corresponding synthetic speech utterance is obtained. This TTS audio file is output to Box 3115. This means that a speaker in Box 3115 is activated to play back the synthetic speech utterance based on the TTS audio file. The TTS audio file serves as a reference for the speaker to perform the appropriate articulation.

[0048] Box 3120 captures a measurement data stream. This data stream is captured using an articulatory measurement modality for a speaker articulation associated with the synthetic speech utterance output in Box 3115. In other words, the speaker is prompted to imitate or articulate the reproduced synthetic speech utterance through appropriate vocal tract movements.

[0049] Optionally, another audio file could be recorded in box 3120, which reproduces an artificial speech utterance of the speaker (substitute voice audio file). The artificial speech utterance can be generated using a substitute voice, such as an electrolarynx, a shunt valve (also called a voice prosthesis), or a ructus voice. An electrolarynx, sometimes called a "throat back," is a medical device used by people who have lost at least part of their vocal apparatus (usually due to laryngeal cancer) to produce speech. A battery-operated handheld device is pressed against the skin under the lower jaw and generates vibrations that enable speech. Other variations include a tube placed in the mouth that provides acoustic vibrations shaped by the vocal tract.A shunt valve is a placeholder with a small valve, positioned between the trachea and esophagus in a so-called fistula tract. It should be understood that such a substitute voice can be used during Box 3005, but is no longer needed during Box 3015. In other words, the substitute voice is used solely for generating training data; however, once enough training samples are available in the training dataset to train the machine-learned speech synthesis model, the machine-learned speech synthesis model can then be trained. Subsequently, speech synthesis can be performed using the articulatory measurement modality, thus rendering the substitute voice unnecessary.

[0050] Box 3125 checks whether the measurement data stream previously measured in Box 3120 is included in the training dataset. There are different ways to implement this check in Box 3125.

[0051] For example, a metric can be determined that is indicative of the synchronicity of the speaker's articulation (captured by the measurement data stream recorded in Box 3120) with the corresponding synthetic speech output in Box 3115. This metric can be determined, for instance, based on a comparison of the duration of the measurement data stream and the TTS audio file. More generally, the metric can be determined based on both the measurement data stream and the TTS audio file. Alternatively, if a substitute voice audio file is available, it could be taken into account. In this case, the metric could be determined based on a comparison of the substitute voice audio file with the TTS audio file.If the measure indicates a sufficient degree of synchronicity, the corresponding measurement data stream can be included in the training data set (“Yes” branch from Box 3125); otherwise, the measurement data stream can be discarded (“No” branch from Box 3125).

[0052] In another implementation variant of Box 3125 (which can be used alternatively or additionally to a synchronicity measure), the replacement voice audio file is transcribed, thus obtaining an additional text body. This additional text body can be compared with the text body from the current iteration 3198 of Box 3105, that is, with the text body that was passed to the TTS model for generating the TTS audio file. The transcription can be automated using a suitable text recognition model. For example, it can be checked whether all phonemes or all words were articulated. For example, text similarity can be determined.

[0053] Next, the "No" branch on box 3125 is described. The "No" branch means that the current measurement data stream is discarded; then, optionally, box 3130 is executed.

[0054] In Box 3139, control data for the TTS model can optionally be adapted to the specific speaker. This control data can be adjusted, in particular, based on the synchronicity measure. The control data can be adjusted to achieve improved synchronicity when, in the next iteration 3199 of Box 3110, the TTS model is re-inferred using the adapted control data to generate a customized synthetic utterance with adjusted tempo, rhythm, and / or pauses. The control data can also be chosen so that the temporal dynamics of a subsequent synthetic utterance generated in the next iteration 3199 of Box 3110 using the TTS model more closely match the temporal dynamics of the speaker's measured articulation from the previous iteration 3199 of Box 3120.This means that, by means of the control data 3130, a deviation in the time dynamics of the synthetic speech utterance output in the current iteration 3199 from box 3115 and the articulation of the speaker measured in box 3120 in the current iteration 3199 is reduced.

[0055] Next, the "Yes" branch from box 3125 will be described.

[0056] Optionally, Box 3145 is executed: Sometimes, even after several iterations (3199) of the same text body, with potentially adjusted speech rates for the synthetic speech utterance, a residual asynchronicity may remain between the articulation captured by the measurement data stream and the corresponding synthetic speech utterance reproduced by the TTS audio file. Typically, this is not a global asynchronicity, where, for example, the entire articulation (from beginning to end) is performed slower than the corresponding synthetic speech utterance; rather, it involves timing deviations for individual phonemes or words. For example, it would be conceivable to reduce such residual timing deviations by a suitable algorithm applied to the audio file. For instance, a so-called "Dynamic Time Warping" algorithm could be used.Alternatively or additionally to such automatic post-processing of the audio file, it may also be desirable to use a loss function robust against such time deviations when subsequently training the machine-learned speech synthesis model. This could be, for example, a so-called sequence-length-invariant loss function, such as the Connectionist Temporal Classification Loss (CTC). For example, residual local asynchronies can be compensated for in Box 3145; that is, local deviations in the speed of the synthetic speech utterance compared to the articulation represented by the measurement data stream. In Box 3145, a residual asynchrony can be removed by adjusting the time dynamics of the audio file and / or the measurement data stream depending on the measure previously determined in Box 3125.Typically, such digital post-processing of the audio file and / or the measurement data stream can remove even a relatively small residual asynchronicity. Therefore, if the asynchronicity is relatively large, a further iteration 3199 is executed; whereas if the audio files and the measurement data stream are already comparatively synchronous, box 3145 is executed.

[0057] Box 3150 is then executed; this means that a corresponding training sample, which includes the TTS audio file and / or the replacement voice audio file from the last iteration 3199 as well as the measurement data stream from the last iteration 3199 of Box 3120, is included in the training data set.

[0058] Box 3155 then checks whether another training sample is needed; if so, another iteration 3198 of Box 3105 is performed. Otherwise, Box 3005 terminates.

[0059] Fig. Figure 2 is an example implementation. Several variations are conceivable. For example, it would be possible to include further termination criteria for the loops defined by iterations 3199 in Box 3125.

[0060] For example, Box 3125 could also check whether a certain number of iterations (3199) has been reached in the current iteration (3198). In this case, Box 3145 could be executed.

[0061] Fig.Figure 3 schematically illustrates a system comprising a data processing device 60 and a sensor device 66. The sensor device 66 is located on or near a speaker 20 and is configured to provide a measurement data stream for the speaker's vocal tract. The sensor device 66 implements an articulatory measurement modality that detects movement of one or more components of the speaker's vocal tract. For example, the sensor device 66 could include one or more radar or ultrasonic antennas. These could be attached to a skin surface in the cheek area of ​​the speaker 20. However, these are only examples, and various sensor modalities are known that can be used.

[0062] A processor 61 of the data processing device 60 receives and records the measurement data stream via a corresponding interface 63. The processor 61 can also load and execute program code from memory 62. When the processor 61 executes the program code, this causes the processor 61 to implement techniques as described herein, for example, in connection with Fig. 1 or Fig.2. For example, the processor 61 can, based on the program code from memory 62, populate a training data set for training a machine-learned speech synthesis model, initiate corresponding training of the machine-learned speech synthesis model (e.g., by sending the training data set to a cloud server via a communication interface 64, where the training is performed), or execute a TTS model, play a TTS audio file to a loudspeaker 68 via a communication interface 64, or receive an audio file via the communication interface 64 that reproduces an artificial speech utterance of the speaker 20 (the artificial speech utterance may be generated by means of a substitute voice).

[0063] In summary, the preceding sections described techniques that enable speaker-specific adaptation of a machine-learned speech synthesis model, even for voiceless speakers. This allows, for example, anatomically and / or technically induced changes in the measurement data streams to be taken into account. Such technical changes can be caused, for instance, by a variation in sensor position. This prevents a deterioration in the machine-learned speech synthesis model's function due to so-called data drift. Repeated adaptation of the machine-learned speech synthesis model for each voiceless speaker, or even for a specific voiceless speaker, becomes possible.

[0064] The techniques described above use synthetic audio speech samples as a reference for otherwise missing speech references for a voiceless speaker. Continuous feedback between the speaker and the TTS model is implemented, resulting in quasi-synchronous speech references. Optionally, a separate substitute voice can be used during the acquisition of training data to record the speech references. For example, speech rate, timbre, etc., can be adjusted in several iterations. This eliminates the need for articulatory measurement signals from the substitute speaker.

[0065] Naturally, the features of the embodiments and aspects of the invention described above can be combined with one another. In particular, the features can be used not only in the combinations described, but also in other combinations or individually, without leaving the scope of the invention.

Claims

[1] Computer-implemented method for speaker-specific adaptation of a machine-learned speech synthesis model, which translates measurement data streams of an articulatory measurement modality into synthetic speech utterances, the procedure includes: - Populating (3005) a training dataset with speaker-specific training samples, and - Initiate (3010) training of the machine-learned speech synthesis model based on the training dataset, where the loading (3005) of the training data set is done iteratively, where each iteration (3199) of populating the training dataset includes: - for a given text body, generating (3110) a corresponding audio file using a text-to-speech model for a synthetic speech utterance that corresponds to the corresponding text body, - based on the corresponding audio file, controlling (3115) a loudspeaker to play back the synthetic speech utterance, - Acquisition (3120) of a corresponding measurement data stream using the articulatory measurement modality for an articulation of the speaker associated with the synthetic speech utterance, and - optionally include (3150) the corresponding measurement data stream in the training data set. [2] Computer-implemented method according to claim 1, where each iteration of populating the training dataset includes: - Determining a corresponding measure that is indicative of the synchronicity of the speaker's articulation with the corresponding synthetic speech utterance, where the measurement data stream is selectively included in the training data set depending on the corresponding measure. [3] Computer-implemented method according to claim 2, where the measure value is determined in a specific iteration based on the corresponding measurement data stream and the corresponding audio file. [4] Computer-implemented method according to claim 2, wherein the measure is determined in a specific iteration based on the corresponding measurement data stream and a further audio file that reproduces an artificial speech utterance generated by means of a substitute voice for articulation. [5] Computer-implemented method according to any one of the preceding claims, where the corresponding audio file is generated in at least one iteration using the text-to-speech model depending on control data, where the control data specify values ​​for at least one parameter associated with speech rate, where the control data is determined speaker-specifically. [6] Computer-implemented method according to claim 5 and according to any one of claims 2 to 4, wherein the method further comprises: - Determining (3130) the control data as a function of the measures from one or more previous iterations (3199). [7] Computer-implemented method according to claim 5 or 6, wherein the at least one parameter is selected from the following group: Tempo, rhythm, pauses. [8] Computer-implemented method according to any of the preceding claims, wherein at least one of the iterations further comprises: - Selecting (3105) the appropriate text body from a multitude of candidates based on a phoneme sequence of one or more text bodies from one or more previous iterations (3198, 3199). [9] Computer-implemented method according to one of the preceding claims, wherein at least one of the iterations of populating the training data set further comprises: - if the measurement data stream is not included in the training data set, discard the corresponding audio file and the corresponding measurement data stream. [10] Computer-implemented method according to one of the preceding claims, wherein at least one of the iterations of populating the training data set further comprises: - optionally adjust (3145) a time dynamic of at least one of the corresponding audio file or measurement data stream. [11] Computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Setting the text-to-speech model depending on prosody control data associated with the speaker. [12] Computer-implemented method according to one of the preceding claims, wherein the measurement data stream is included in the training data set as a training sample together with the audio file and / or a further audio file associated with articulation, which reproduces an artificial speech utterance generated by means of a substitute voice. [13] Computer-implemented method according to one of the preceding claims, wherein at least one of the iterations of populating the training data set further comprises: - Receiving another audio file that reproduces an artificial speech utterance generated by means of a substitute voice for articulation, whereby the measurement data stream is optionally included in the training data set depending on the other audio file. [14] Computer-implemented method according to claim 13, where at least one of the iterations of populating the training dataset still includes: - Performing a comparison between the text body associated with the audio file and a further text body obtained by transcribing the additional audio file, where the measurement data stream is optionally included in the training data set depending on a result of the comparison. [15] Electronic data processing device comprising a processor and a memory, wherein the processor is configured to load and execute program code from the memory, wherein the execution of the program code causes the processor to perform the following steps: - Populating (3005) a training dataset with speaker-specific training samples, and - Initiating (3010) the training of a machine-learned speech synthesis model based on the training dataset, wherein the provisioning (3005) of the training dataset is iterative, with each iteration comprising (3199): - for a given text body, generating (3110) a corresponding audio file using a text-to-speech model for a synthetic speech utterance that corresponds to the corresponding text body, - based on the corresponding audio file, controlling (3115) a loudspeaker to play back the synthetic speech utterance, - Acquisition (3120) of a corresponding measurement data stream using an articulatory measurement modality for an articulation of the speaker that is associated with the synthetic speech utterance, - optionally include (3150) the corresponding measurement data stream in the training data set. [16] Electronic data processing device according to claim 15, wherein the execution of the program code causes the processor to execute a method according to any one of claims 1 to 14.

Images