Method for conditioning speaker information in TTS model based on lightweight convolution and a system for executing the same
The SDLC block in the speech synthesis model addresses performance degradation and inefficiency by directly applying speaker embeddings as kernel weights, enhancing speech synthesis quality and similarity.
Patent Information
- Application Number
- KR1020240159654
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-11-11
AI Technical Summary
Existing methods for conditioning speaker information in speech synthesis models using speaker embeddings applied to CNN kernel weights face performance degradation, inefficiency, and complex reconstruction of kernel weights, particularly when applied to the decoder.
A method employing a Speaker Dynamic Lightweight Convolution (SDLC) block with a first and second speaker layer to condition speaker information in the kernel parameters, text domain, and spectrogram domain of the convolution, applied to both the encoder and decoder of the speech synthesis model.
Improves the similarity and quality of synthesized speech by directly applying speaker embeddings as kernel weights, reducing variance and simplifying the process, while maintaining superior performance and versatility.
Smart Images

Figure 112024123904874-PAT00002_ABST
Abstract
Description
Technology Field
[0001] The disclosed invention relates to a method for conditioning speaker information in a speech synthesis model based on lightweight convolution, wherein speaker embeddings are conditioned in the kernel parameters, text domain, and spectrogram domain of the convolution so that speaker information is reflected in the speech synthesis model, and a system for executing the same. Background Technology
[0002] Recent advancements in speech synthesis technology have significantly improved the quality and naturalness of generated synthesized speech, reaching a level where it is possible to synthesize voices that are more natural, resembling human voices. This speech synthesis technology is being utilized in various fields, including automatic response services (ARS), AI (artificial intelligence) English tutors, content dubbing, and the treatment of speech disorders. Consequently, there is a continuously increasing need for speech synthesis models capable of providing diverse voice options and for custom voice generation technologies tailored to individual preferences.
[0003] A text-to-speech (TTS) system is a system that synthesizes speech based on given input text. A TTS system is broadly composed of a feature generation process and a waveform transformation process. The feature generation process involves taking text as input using an acoustic model to generate intermediate features such as mel-spectrograms or MFCCs (mel-frequency cepstral coefficients). The waveform transformation process converts these intermediate features into a raw waveform form using a vocoder model.
[0004] In a speech synthesis system, the process of replicating the voice of a new speaker that is not in the training dataset is called voice cloning. Among voice cloning techniques, the process of replicating and synthesizing voice without fine-tuning using only a few seconds of reference voice is called zero-shot voice cloning.
[0005] For speech synthesis models using zero-shot speech replication, the method of injecting speaker information (conditioning) is important. Methods that have been widely used include simply adding speaker information to the output of the TTS model's encoder, or conditioning speaker information by using a feature-wise linear modulation layer that performs conversion between elements.
[0006] SC-CNN (Speaker Conditioning Convolutional Neural Network) is a method for conditioning speaker information in a TTS model by applying speaker embeddings, which represent speaker information, to the kernel weights of a CNN. However, the method of conditioning speaker information in a speech synthesis model by applying speaker embeddings to the kernel weights of a CNN has the problem that performance degrades when speaker information is added to the decoder of the TTS model. Furthermore, since the method of conditioning speaker information in a speech synthesis model by applying speaker embeddings to the kernel weights of a CNN uses general 1D depth-wise separable convolution, directly applying speaker embeddings as kernel weights is not efficient. Additionally, the method of conditioning speaker information in a speech synthesis model by applying speaker embeddings to the kernel weights of a CNN has the problem of requiring complex reconstruction of kernel weights by predicting values reflecting speaker embeddings through a kernel predictor. The problem to be solved
[0007] The disclosed embodiment for solving these problems relates to a method for conditioning speaker information in a speech synthesis model based on lightweight convolution, which can improve the quality of the speech synthesized through the speech synthesis model and the similarity of the replicated speech by conditioning speaker information in the speech synthesis model, and a system for executing the same. means of solving the problem
[0008] A method for conditioning speaker information in a speech synthesis model based on lightweight convolution according to a disclosed embodiment and a system for executing the same comprises: a method for conditioning speaker information in a speech synthesis model based on lightweight convolution, performed by a system including at least one processor, comprising: a data acquisition process for acquiring a speaker embedding that reflects the timbre of the speaker in a target speech; and a conditioning process for conditioning an output for the speaker embedding based on a lightweight neural network architecture including a plurality of lightweight convolution blocks; wherein the conditioning process adds an SDLC block including a first speaker layer and a second speaker layer to perform a Speaker Dynamic Lightweight Convolution (SDLC) operation that supplies the speaker embedding to the kernel parameters of the convolution of the lightweight convolution block, a text domain, and a text domain and a spectrogram domain.
[0009] Alternatively, the first speaker layer adds speaker information to the text domain when the SDLC block is applied to the encoder and adds speaker information to the spectrogram domain when the SDLC block is applied to the decoder, and the second speaker layer directly applies speaker information to the kernel parameters of the convolution.
[0010] Alternatively, the conditioning process comprises: a first step in which the acquired speaker embedding passes through a first speaker layer and a second speaker layer, respectively; a second step in which the speaker embedding that has passed through the first speaker layer sets the dimension to match the input dimension of the SDLC block and expands the speaker embedding based on the input sequence length; a third step in which the speaker embedding that has passed through the second speaker layer performs dimension reduction based on the kernel size and the number of channels, and then performs shape change into a kernel shape; and a fourth step in which the shape-changed speaker embedding is applied as a kernel weight to condition speaker information to the speech synthesis model.
[0011] Alternatively, the fourth step further includes the step of summing the speaker embeddings and input data obtained from the second step, linearly projecting the summed data into two, multiplying the two linear projections by component in Gated Linear Units (GLUs), and then performing SDLC operations and layer normalization with kernel weights.
[0012] Alternatively, the SDLC block is added to the encoder or decoder of the speech synthesis model, respectively.
[0013] A system for conditioning speaker information in a speech synthesis model based on lightweight convolution according to a disclosed embodiment comprises: a memory for storing a plurality of neural network models including a speech synthesis model; and a processor for training the neural network models and performing a speech synthesis function for input data input through the trained neural network models. The processor acquires a speaker embedding that reflects the timbre of the speaker in a target speech and conditions the output of the speaker embedding based on a lightweight neural network architecture that includes a plurality of lightweight convolution blocks. The lightweight convolution blocks include an SDLC block comprising a first speaker layer and a second speaker layer to perform a Speaker Dynamic Lightweight Convolution (SDLC) operation that supplies the speaker embedding to the kernel parameters, text domain, and spectrogram domain of the convolution.
[0014] A computer program according to a disclosed embodiment is a computer program stored on a computer-readable storage medium, wherein, when executed on one or more processors, the computer program performs operations for conditioning speaker information in a speech synthesis model based on lightweight convolution, the operations include: a data acquisition operation for acquiring a speaker embedding that reflects the speaker's timbre in a target speech; and a conditioning operation for conditioning an output for the speaker embedding based on a lightweight neural network architecture comprising a plurality of lightweight convolution blocks, wherein the conditioning operation adds an SDLC block comprising a first speaker layer and a second speaker layer to perform a Speaker Dynamic Lightweight Convolution (SDLC) operation that supplies the speaker embedding to the kernel parameters, text domain, and spectrogram domain of the convolution of the lightweight convolution block. Effects of the invention
[0015] A method for conditioning speaker information in a speech synthesis model based on lightweight convolution according to a disclosed embodiment and a system for executing the same can reflect speaker information in a lightweight neural network model through a speaker dynamic lightweight convolution (SDLC) block for conditioning speaker embeddings in the kernel parameters, text domain, and spectrogram domain of the convolution of the speech synthesis model, and can improve the similarity between the original speech of a target speaker and the synthesized speech through such a speech synthesis model, and can increase versatility by simultaneously applying the SDLC block to the encoder and decoder of the speech synthesis model. Brief explanation of the drawing
[0016] FIG. 1 is a control block diagram of a method for conditioning speaker information in a speech synthesis model based on disclosed lightweight convolution and a system for executing the same. FIG. 2 is a flowchart illustrating a method for conditioning speaker information in a speech synthesis model based on lightweight convolution according to a disclosed embodiment. FIG. 3 is a flowchart showing in detail the conditioning process of a method for conditioning speaker information in a speech synthesis model based on lightweight convolution according to a disclosed embodiment. FIG. 4 is a diagram illustrating the configuration of an SDLC block based on a lightweight neural network architecture according to a disclosed embodiment. FIG. 5 is a diagram illustrating the SDLC operation process of an SDLC block based on a lightweight neural network architecture according to a disclosed embodiment. Figure 6 is a diagram that specifically explains the results of comparing the performance of a method for conditioning speaker information in a speech synthesis model based on the disclosed lightweight convolution with that of a conventional conditioning method. FIG. 7 is a diagram showing the results of comparing the speech processing speeds of an SDLC block based on a lightweight neural network architecture according to the disclosed embodiment and an SC-CNN. FIG. 8 is a graph showing the average variation of kernel weights in the last layer of the encoder of the SDLC block and SC-CNN based on the lightweight neural network architecture according to the disclosed embodiment. Specific details for implementing the invention
[0017] Throughout the specification, the same reference numerals refer to the same components. This specification does not describe all elements of the embodiments, and general content in the art to which the invention pertains or content that overlaps between embodiments is omitted.
[0018] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.
[0019] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.
[0020] Singular expressions include plural expressions unless there is an obvious exception in the context.
[0021] In addition, terms such as "~part," "~unit," "~block," "~part," and "~module" may refer to a unit that processes at least one function or operation. For example, the above terms may refer to at least one piece of hardware such as an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit), at least one piece of software stored in memory, or at least one process processed by a processor.
[0022] The symbols attached to each step are used to identify each step and do not indicate the order of the steps relative to one another; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.
[0023] Hereinafter, with reference to the attached drawings, embodiments relating to a method for conditioning speaker information in a speech synthesis model based on lightweight convolution and a system for executing the same will be described in detail.
[0024] Text-to-Speech (TTS) is a neural network that synthesizes speaker speech from text, and the latest TTS models can synthesize speaker speech from training datasets with high quality.
[0025] When a TTS model is trained on a multi-speaker dataset while conditioning it with speaker embeddings containing the speaker's voice features derived from the speaker's voice data, a single TTS model can synthesize the voices of multiple speakers present in the dataset. However, since it is difficult to acquire large amounts of voice data for target speakers, the TTS model must be able to generate voices of speakers that are not present in the training dataset.
[0026] Generally, training datasets used in TTS models utilize large amounts of voice data recorded by voice actors in noise-free environments and labeled text data; however, acquiring large quantities of noise-free voice data from target speakers is practically very difficult and inevitably costly. Consequently, to train a TTS model centered on target speakers, the model can be pre-trained using a dataset of a large number of random speakers, followed by fine-tuning using a small amount of voice data from the target speakers.
[0027] Zero-shot speaker application techniques that require no fine-tuning or a small dataset obtainable from target speakers are being actively researched, and the replication of the voice of a target speaker—that is, a seen speaker—that is, a speaker not present in such training datasets is called zero-shot voice cloning. Here, a speaker present in the training dataset is called a seen speaker, and a speaker not present in the training dataset is called an unseen speaker.
[0028] Zero-shot voice replication technology aims to generate a voice similar to a target speaker using only a few seconds of their voice data, and to achieve this, speaker representation and speaker conditioning technologies are crucial.
[0029] A method for conditioning speaker information in a speech synthesis model based on lightweight convolution in the disclosed embodiment and a system for executing the same provide speaker conditioning technology optimized for zero-shot speaker applications.
[0030] FIG. 1 is a control block diagram of a method for conditioning speaker information in a speech synthesis model based on lightweight convolution in a disclosed embodiment and a system for executing the same.
[0031] The method for conditioning speaker information in a speech synthesis model based on lightweight convolution in the disclosed embodiment and the system (1) for executing the same can be implemented as a computer or portable terminal (hereinafter user terminal, 3) capable of connecting to a communication network such as the Internet. Here, the computer includes, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser, and the portable terminal can be implemented as, for example, any type of handheld-based wireless communication device such as a smartphone, etc., as a wireless communication device that ensures portability and mobility.
[0032] Referring to FIG. 1, a user terminal (3) may include a communication interface (11) for receiving various data, such as an artificial intelligence model or training data required for speech synthesis, from an external server (2); a memory (12) for storing the received training data and artificial intelligence model; an input unit (13) for receiving text to be converted into speech or other various user input commands; a processor (10) for controlling the overall system; and an output unit (14) for outputting the synthesized speech performed by the processor (10) as sound. However, since FIG. 1 is merely an example, the user terminal (3) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the user terminal (3).
[0033] Specifically, the communication interface (11) may include one or more components that enable communication with an external communication network, and may include, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.
[0034] The short-range communication module may include various short-range communication modules that transmit and receive signals using a wireless communication network at short range, such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module.
[0035] Wired communication modules may include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).
[0036] In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, the wireless communication module may include wireless communication modules that support various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution).
[0037] The memory (12) may store an artificial intelligence model required to implement a method for conditioning speaker information in a speech synthesis model based on a lightweight convolution in the disclosed embodiment, an algorithm required for the operation of the processor (10), or a program for implementing the algorithm. To this end, the memory (12) may be implemented as at least one of a non-volatile memory device such as a cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory, a volatile memory device such as RAM (Random Access Memory), or a storage medium such as a hard disk drive (HDD) and a CD-ROM, but is not limited thereto.
[0038] The input unit (13) can receive various input commands. Specifically, the input unit (13) may include a microphone that receives voice that speaks into a video containing text as subtitles, in addition to receiving input text. Furthermore, the input unit (13) may include hardware devices such as various buttons, switches, pedals, keyboards, mice, trackballs, various levers, handles, or sticks for receiving execution commands required for the operation of the processor (10). Additionally, the input unit (13) may include a GUI (Graphical User Interface), i.e., a software device such as a touch pad, for user input commands. The touch pad may be implemented as a touch screen panel (TSP) and form a layered structure with the display of the output unit (14).
[0039] The output unit (14) may include a speaker that outputs the recognized voice as sound, as well as a display for outputting the learning results or inference results of an artificial intelligence model. The display may be provided as a Cathode Ray Tube (CRT), Digital Light Processing (DLP) panel, Plasma Display Panel, Liquid Crystal Display (LCD) panel, Electro Luminescence (EL) panel, Electrophoretic Display (EPD) panel, Electrochromic Display (ECD) panel, Light Emitting Diode (LED) panel, or Organic Light Emitting Diode (OLED) panel, but is not limited thereto.
[0040] The processor (10) can fine-tune or train an artificial intelligence model stored in memory (12) and perform speech synthesis functions using the trained artificial intelligence model for input text.
[0041] Generally, speech synthesis models utilize both a prediction mode and a generation mode. They convert the input text into a spectrogram using either an autoregressive or non-autoregressive method, and then convert the spectrogram into a speech waveform. FastSpeech2 is typically used as the baseline for the prediction model, while the generation model primarily consists of text encoder and decoder parts.
[0042] The processor (10) adds an SDLC block to the generation model of the speech synthesis model to enable speaker information conditioning.
[0043] Specifically, when the processor (10) adds SDLC blocks to both the encoder and the decoder, it allows speaker information to be reflected in the lightweight neural network model by conditioning the kernel parameters of the text domain, the spectrogram domain, and the convolution. Here, the spectrogram domain refers to the hidden states of the text encoder, the text domain, and the hidden states of the decoder part that predicts the Mel spectrogram.
[0044] A specific method for a processor (10) to perform speech synthesis by a speech synthesis model or to condition speaker information in a speech synthesis model based on lightweight convolution is described later through other drawings below.
[0045] Meanwhile, the processor (10) may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of such data processing devices embedded in hardware may include, but are not limited to, processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a Graphics Processing Unit (GPU), and a neural Processing Unit (NPU). The processor (10) may be provided in multiple units.
[0046] The hardware configurations provided in the user terminal (10) can exchange data and signals via Network Termination (NT) of a digital network such as ISDN (Integrated Services Digital Network).
[0047] Recently, speech synthesis models are not only capable of synthesizing the voices of speakers in a training dataset with high quality, but if speaker embeddings (vectors containing speaker information) are conditioned and the model is trained on a multi-speaker dataset, it is possible to synthesize the voices of multiple speakers in the multi-speaker dataset using a single model.
[0048] In order to practically use such speech synthesis models, it is necessary to be able to generate the voices of speakers not present in the training dataset, as it is difficult to obtain large quantities of the target speaker's voice. Generally, datasets used in speech synthesis models consist of large volumes of voice recordings and labeled text recorded by voice actors in noise-free environments; however, obtaining large quantities of clean voices of target speakers is practically very difficult or costly.
[0049] Traditionally, speech synthesis models are pre-trained using a large dataset of random speakers, followed by fine-tuning using a small amount of target speaker voice data. However, zero-shot speaker application technology, which eliminates the need for both a small dataset of target speakers and a fine-tuning process, is being actively researched. Zero-shot speaker application technology is also referred to as zero-shot voice cloning because it replicates the voice of an unseen speaker—that is, a target speaker—that is not present in the training dataset. Since speaker representation methods and speaker conditioning methods are important in zero-shot voice cloning technology, the present invention, unlike existing FiLM and SC-CNN, adds an SDLC block to allow speaker embeddings to be conditioned in the convolution kernel parameters, text domain, and spectrogram domain.
[0050] FIG. 2 is a flowchart illustrating a method for conditioning speaker information in a speech synthesis model based on lightweight convolution according to a disclosed embodiment.
[0051] The method performed by the system (1) for conditioned speaker information in a speech synthesis model based on the disclosed lightweight convolution can be applied to deep learning-based TTS models such as Transformers, CNNs, and RNNs. In this case, the Transformer may include a neural network that converts an input sequence into a sequence of embedding vectors called hidden states or contexts in an encoder, and in a decoder, iteratively generates output sequences one at a time using the hidden states of the encoder and outputs feature vectors for searching for words (or strings). The text sequence output from such a Transformer is converted into a sequence of probability vectors at each time step through a linear and a softmax function of a classifier. The softmax function takes an arbitrary vector as input and returns an output in the form of a discrete probability distribution, and is a function that converts the result so that it can be interpreted as a probability, classifying it into a class with a high probability.
[0052] The above method obtains a speaker embedding that reflects the speaker's timbre in the target speech through a speaker encoder, etc., during the data acquisition process (S10).
[0053] The conditioning process (S20) conditions the output for the speaker embedding based on a lightweight neural network architecture that includes a plurality of lightweight convolution blocks. The lightweight neural network model includes a plurality of lightweight convolution blocks and includes an SDLC block (100) that includes a first speaker layer and a second speaker layer to perform Speaker Dynamic Lightweight Convolution (SDLC) operations.
[0054] Here, the SDLC block (100) is added to the encoder and decoder of the TTS model, respectively, and the speaker layer adds speaker information to the input of the SDLC block. The first speaker layer is a layer that adds speaker information to the text domain when the SDLC block is applied to the encoder, and to the spectrogram domain when the SDLC block is applied to the decoder, respectively, and the second speaker layer is a layer that directly applies speaker information to the kernel parameters of the convolution.
[0055] FIG. 3 is a flowchart illustrating in detail the conditioning process of a method for conditioning speaker information in a speech synthesis model based on lightweight convolution according to a disclosed embodiment. FIG. 4 is a diagram explaining the configuration of an SDLC block based on a lightweight neural network architecture according to a disclosed embodiment, and FIG. 5 is a diagram explaining the SDLC operation process of an SDLC block based on a lightweight neural network architecture according to a disclosed embodiment.
[0056] The conditioning process (S20) according to the disclosed embodiment may include steps for conditioning a speaker embedding representing speaker information to a TTS model through an SDLC block (100) including a first speaker layer (110) and a second speaker layer (120).
[0057] When the acquired speaker embedding passes through the first speaker layer (110) (S21), the system (1) sets the dimension of the speaker embedding that passed through the first speaker layer (110) to match the input dimension of the SDLC block (S22), expands the speaker embedding to match the input sequence length, and then adds the expanded speaker embedding to the input data (S23, S24).
[0058] The data summed from the speaker embedding and the input data is linearly projected into two (linear) (130), and the two linear projections are multiplied by their components in Gated Linear Units (GLU) (140) and then SDLC (150) operations are performed with kernel weights.
[0059] As illustrated in FIG. 5, the system (1) performs dimensionality reduction on the speaker embeddings that have passed through the second speaker layer (120) based on the value obtained by multiplying the kernel size and the number of channels (or number of heads) (S21, S25), and performs reshape into a kernel shape (S26).
[0060] As shown in Fig. 5, in the case of 1D lightweight convolution, the number of channels (or heads) can be adjusted to a much lower number of channels compared to general 1D depth-wise convolution. This lightweight convolution characteristic is useful when directly applying speaker embeddings as kernel weights, and it demonstrates superior performance by being faster at handling the overall context in the TTS model than general convolution.
[0061] The system (1) directly applies the shape-changed speaker embeddings as kernel weights and conditions the speaker information in the speech synthesis model through SDLC operations and layer normalization (S27). In the layer normalization (or layer norm) process, the mean and standard deviation are calculated for each data sample to perform normalization.
[0062] The system (1) uses a lightweight CNN that can adjust the number of channels to be small so that speaker embeddings are directly applied as kernel weights using SDLC blocks (100), but since the structure of the lightweight CNN and the type of neural network described above are merely examples, the neural network model according to one embodiment of the present invention can be configured in various ways based on the example described above.
[0063] In the case of SC-CNN, since it is designed using inductive bias—which suggests that the phonetic domain in TTS models is highly related to speaker characteristics (or speaker information)—it is used only in the encoder and not applied to the decoder. In fact, SC-CNN has the disadvantage that performance actually degrades when the phonetic domain is applied to the decoder.
[0064] However, in the present invention, since the SDLC block (100) that performs SDLC operations is applied to both the encoder and the decoder, speaker characteristics are supplied to the kernel parameters of the convolution, the text domain, and the spectrogram domain through the speaker layer, thereby solving the problem of performance degradation when SC-CNN is applied to the decoder. Therefore, a TTS model to which a lightweight neural network architecture with an added SDLC block is applied compared to SC-CNN has superior performance by reflecting speaker embeddings.
[0065] In addition, the present invention reduces the variance of kernel parameters in convolution, thereby reducing discrepancy between training and inference, and can speed up the process by allowing speaker information to be directly applied to kernel weights. In the case of voice cloning, since the conditioning variables of speaker information (or speaker embeddings) predicted from unseen speakers during inference are highly likely to have a different distribution from the conditioning variables derived from speaker information of seen speakers used during training, the distribution of conditioning variables when speaker embeddings are reflected in the TTS model must not be too complex. To this end, SC-CNN must predict two separate components from speaker information and reconstruct kernel weights from the predicted components. However, in the present invention, speaker embeddings can be directly applied as kernel weights using only the second speaker layer (120) with weights within the SDLC block, which has the advantage of simplifying the overall process and relaxing the distribution of conditioning variables to improve similarity.
[0066] The present invention uses PyTorch, a Python-based deep learning toolkit, to condition speaker embeddings into a TTS model and uses a GPU RTX4090 for training the new TTS model. For the training and execution of the TTS model, the public multi-speaker datasets LibriTTS train-clean-360 and VCTK are used. Specifically, both the LibriTTS dataset and the VCTK dataset are used to pre-train the TTS model, and the TestRol dataset is used for reference speech of invisible speakers for zero-shot speech replication. In this way, by utilizing a large number of training datasets and pre-training with more speakers, the performance of zero-shot speech replication can be improved.
[0067] The TTS model uses the open-source code FastSpeech2 as its baseline, and the speaker embeddings use style embeddings extracted from StyleSpeech's mel-style encoder. Once the text is converted into an intermediate mel spectrogram using the TTS model, it can be synthesized into speech using a vocoder with a HiFi-GAN structure.
[0068] By using validation mel-spectrogram loss per training step to compare the performance of a method for conditioning speaker information in a speech synthesis model based on lightweight convolution with existing conditioning methods, the quality of synthesized speech of invisible speakers not present in the training dataset can be objectively compared. In this case, FastSpeech2 is used as the TTS model, and style embeddings extracted from a mel style encoder can be used for speaker embeddings.
[0069] FIG. 6 is a diagram that specifically explains the results of comparing the performance of a method for conditioning speaker information in a speech synthesis model based on the disclosed lightweight convolution with that of a conventional conditioning method. FIG. 7 is a diagram showing the results of comparing the speech processing speed of an SDLC block based on a lightweight neural network architecture according to the disclosed embodiment and an SC-CNN, and FIG. 8 is a graph showing the average variation of kernel weights in the last layer of the encoder of an SDLC block based on a lightweight neural network architecture according to the disclosed embodiment and an SC-CNN.
[0070] Figures 6 through 8 show the Mean Opinion Score (MOS) and Similarity Mean Opinion Score (SMOS) of replicated speech of a seen speaker present in the training dataset and a replicated speech of an unseen speaker not present in the training dataset, within a 95% confidence interval. Additionally, for objective metrics, the speaker embedding cosine similarity (SEC) of the synthesized speech using ECAPA-TDNN and the reference speech used was compared. In Figure 8, when the embeddings of random unseen speakers and seen speakers are conditioned, the kernel size is set to 9 for both. It can be seen that the variation in kernel weights in the SDLC block is lower than that of SC-CNN.
[0071] As shown in Figure 6, it can be seen that FiLM (SALN) is very difficult to convert to the speech style of an unseen speaker with a small number of variables, whereas SDLC is able to convert to a robust and effective speech style using each kernel parameter as a conditioning variable.
[0072] As shown in Figure 7, through the Real Time Factor (RTF) used as a performance metric to measure how much faster the speech synthesis model processes speech in real time, it can be seen that the speech synthesis model with SDLC blocks applied demonstrates faster processing speed and superior performance compared to SC-SNN.
[0073] As shown in Figure 8, when SDLC blocks are applied simultaneously to the encoder and decoder of a speech synthesis model, it can be seen that SDLC blocks have a higher similarity to the replicated speech than SC-CNN because the distribution of kernel weights, i.e., the variability, is low.
[0074] Furthermore, the present invention does not degrade performance even when speaker information is conditioned in the spectrogram domain and added to the decoder, so it can be more versatile and improve performance compared to SC-CNN, which is conditioned only on kernel weights.
[0075] As such, it can be seen at a glance that the present invention demonstrates significantly superior performance compared to existing conditioning methods in terms of the quality of synthesized speech and the similarity of replicated speech using a TTS model conditioned with speaker embeddings.
[0076] The method of conditioning speaker information to a speech synthesis model according to the present invention can be extended and applied not only to speaker embeddings but also to various conditioning embeddings such as pitch embeddings.
[0077] Although FIGS. 2 and FIGS. 3 describe each process as being executed sequentially, this is merely an illustrative explanation of the technical concept of one embodiment of the present invention. In other words, a person skilled in the art to which one embodiment of the present invention belongs can modify and adapt the process in various ways, such as changing the order described in each figure or executing one or more of the processes in parallel, without departing from the essential characteristics of one embodiment of the present invention; therefore, FIGS. 2 and FIGS. 3 are not limited to a chronological order.
[0078] Meanwhile, the processes illustrated in FIGS. 2 and 3 can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. That is, a computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.). In addition, the computer-readable recording medium can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Explanation of the symbols
[0079] 1: System 2: Server 3: User terminal 11: Communication Interface 12: Memory 13: Input section 10: Processor 14: Output section 100: SDLC block
Claims
Claim 1 A method for conditioning speaker information in a speech synthesis model based on lightweight convolution, performed by a system comprising at least one processor, comprising: a data acquisition process for acquiring a speaker embedding that reflects the speaker's timbre in a target speech; A method comprising: a conditioning process for conditioning the output of the speaker embedding based on a lightweight neural network architecture comprising a plurality of lightweight convolution blocks; wherein the conditioning process comprises adding an SDLC block comprising a first speaker layer and a second speaker layer to perform a Speaker Dynamic Lightweight Convolution (SDLC) operation that supplies the speaker embedding to the kernel parameters of the convolution, a text domain, and a spectrogram domain of the lightweight convolution blocks; wherein the first speaker layer adds speaker information to the text domain when the SDLC block is applied to an encoder and adds speaker information to the spectrogram domain when the SDLC block is applied to a decoder, and the second speaker layer directly applies speaker information to the kernel parameters of the convolution. Claim 2 delete Claim 3 A method according to claim 1, wherein the conditioning process comprises: a first step in which the acquired speaker embedding passes through a first speaker layer and a second speaker layer, respectively; a second step in which the speaker embedding that has passed through the first speaker layer sets the dimension to match the input dimension of the SDLC block and expands the speaker embedding based on the input sequence length; a third step in which the speaker embedding that has passed through the second speaker layer performs dimension reduction based on the kernel size and the number of channels, and then performs shape change into a kernel shape; and a fourth step in which the shape-changed speaker embedding is applied as a kernel weight to condition speaker information in the speech synthesis model. Claim 4 A method according to claim 3, wherein the fourth step further comprises the step of summing the speaker embedding obtained from the second step and the input data, linearly projecting the summed data into two, multiplying the two linear projections by component in a Gated Linear Unit (GLU), and then performing SDLC operations and layer normalization together with kernel weights. Claim 5 A method according to claim 1, wherein the SDLC block is added to the encoder or decoder of the speech synthesis model, respectively. Claim 6 A system for conditioning speaker information in a speech synthesis model based on lightweight convolution, comprising: a memory for storing a plurality of neural network models including a speech synthesis model; and a processor for training the neural network models and performing a speech synthesis function for input data input through the trained neural network models; wherein the processor acquires a speaker embedding that reflects the speaker's timbre in a target speech and, based on a lightweight neural network architecture including a plurality of lightweight convolution blocks, conditions the output of the speaker embedding, wherein the lightweight convolution blocks include an SDLC block including a first speaker layer and a second speaker layer to perform a Speaker Dynamic Lightweight Convolution (SDLC) operation that supplies the speaker embedding to the kernel parameters of the convolution, a text domain, and a spectrogram domain, and wherein the first speaker layer adds speaker information to the text domain when the SDLC block is applied to an encoder, and when the SDLC block is applied to a decoder A system in which speaker information is added to the spectrogram domain, and the second speaker layer directly applies the speaker information to the kernel parameters of the convolution. Claim 7 delete Claim 8 A system according to claim 6, wherein the processor passes the speaker embedding through the first speaker layer and the second speaker layer, respectively, sets the dimension of the speaker embedding that has passed through the first speaker layer to match the input dimension of the SDLC block, expands the speaker embedding based on the input sequence length, performs dimension reduction on the speaker embedding that has passed through the second speaker layer based on the kernel size and the number of channels, performs shape change into a kernel shape, and applies the shape-changed speaker embedding as a kernel weight to condition speaker information to the speech synthesis model. Claim 9 In paragraph 6, the SDLC block is added to the encoder or decoder of the speech synthesis model, respectively, in a system. Claim 10 A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for conditioning speaker information in a speech synthesis model based on lightweight convolution, and the operations include a data acquisition operation for acquiring a speaker embedding that reflects the speaker's timbre in a target speech; A computer program comprising: a conditioning operation for conditioning the output of the speaker embedding based on a lightweight neural network architecture including a plurality of lightweight convolution blocks; wherein the conditioning operation comprises adding an SDLC block including a first speaker layer and a second speaker layer to perform a Speaker Dynamic Lightweight Convolution (SDLC) operation that supplies the speaker embedding to the kernel parameters of the convolution, a text domain, and a spectrogram domain of the lightweight convolution blocks; wherein the first speaker layer adds speaker information to the text domain when the SDLC block is applied to an encoder and adds speaker information to the spectrogram domain when the SDLC block is applied to a decoder, and the second speaker layer directly applies speaker information to the kernel parameters of the convolution.
