Apparatus and method for text analysis and speech synthesis

Abstract

Disclosed are an apparatus and a method for text analysis and speech synthesis. The apparatus for text analysis and speech synthesis according to an embodiment includes: a text analysis module for generating a plurality of text chunks by separating input text into utterance units; a text encoding module for generating a plurality of text feature chunks by encoding the plurality of generated text chunks; and a speech synthesis module for generating speech signals on the basis of the plurality of text feature chunks.

Description

Text analysis and speech synthesis device and method

[0001] The disclosed embodiments relate to a technique for analyzing and separating a given text into utterance units and synthesizing speech based on the same.

[0002] Speech synthesis is a technology for generating natural-sounding speech based on text. To generate speech from text, the given text must first be encoded to create the text features necessary for speech synthesis, and then these features must be used to synthesize speech.

[0003] Deep learning-based speech synthesis takes into account the context of the input text, requiring the entire text to be encoded beforehand to synthesize natural-sounding speech. However, this approach incurs excessive delay, especially in real-time speech synthesis.

[0004] The disclosed embodiments are intended to provide a technical means for analyzing a given text into utterance units, separating the same, and synthesizing speech based on the same.

[0005] According to an exemplary embodiment, a text analysis and speech synthesis device is provided, including a text analysis module that divides an input text into utterance units to generate a plurality of text chunks; a text encoding module that encodes the generated plurality of text chunks to generate a plurality of text feature chunks; and a speech synthesis module that generates a speech signal based on the plurality of text feature chunks.

[0006] The text analysis module can segment the input text into utterance units based on one or more of a sentence terminator character, a word-ending suffix, a conjunction, a connecting suffix, or one or more phrase or clause boundaries detected in the input text.

[0007] The text encoding module can generate multiple text feature chunks using one or more of a convolutional neural network, a recurrent neural network, or a transformer.

[0008] The text encoding module can sequentially encode multiple text chunks received from the text analysis module.

[0009] The text encoding module can organize the multiple text chunks received from the text analysis module into one or more batches by considering the lengths of the multiple text chunks, and perform encoding for each batch.

[0010] The text encoding module can encode a specific text chunk using information of one or more previous text chunks for the specific text chunk among multiple text chunks.

[0011] The text encoding module can encode multiple text chunks using context prediction information for each of the multiple text chunks.

[0012] The speech synthesis module can decode multiple text feature chunks to generate multiple speech chunks, and concatenate the generated multiple speech chunks to generate a speech signal.

[0013] The text encoding module may, if the length of a specific text feature chunk among the generated multiple text feature chunks is greater than a threshold value, re-divide the specific text feature chunk into one or more sub-feature chunks and provide the re-divided text feature chunk to the speech synthesis module.

[0014] The speech synthesis module can generate one or more sub-speech chunks by decoding each of one or more sub-feature chunks, and can concatenate the generated one or more sub-speech chunks to generate a speech chunk corresponding to a specific text feature chunk.

[0015] The speech synthesis module can concatenate one or more text feature chunks among a plurality of text feature chunks and decode the concatenated text feature chunks to generate a speech signal.

[0016] The speech synthesis module can sequentially decode a plurality of text feature chunks received from a text encoding module to generate a speech signal.

[0017] The speech synthesis module can generate a speech signal by composing a plurality of text feature chunks into one or more batches by considering the lengths of the plurality of text feature chunks received from the text encoding module and performing decoding for each batch.

[0018] The speech synthesis module can decode a specific text feature chunk using information of one or more previous text feature chunks for the specific text feature chunk among a plurality of text feature chunks.

[0019] The speech synthesis module can generate a speech signal by additionally considering one or more of emotional information and speech style information corresponding to the input text.

[0020] According to another exemplary embodiment, a method for text analysis and speech synthesis is provided, comprising: a step of generating a plurality of text chunks by dividing an input text into utterance units; a step of encoding the generated plurality of text chunks to generate a plurality of text feature chunks; and a step of generating a speech signal based on the plurality of text feature chunks.

[0021] According to the disclosed embodiments, input text is divided into utterance units, which are units read by humans, and each utterance is synthesized into speech, thereby eliminating awkwardness at the boundaries between text chunks. Furthermore, rather than encoding the entire input text and then synthesizing speech based on it, encoding and speech synthesis are performed on a per-text chunk basis, thereby optimizing the time required until speech synthesis begins.

[0022] FIG. 1 is a block diagram illustrating a text analysis and speech synthesis device (100) according to one embodiment of the present invention.

[0023] Figure 2 is a flowchart for explaining a text analysis and voice synthesis method (200) according to one embodiment.

[0024] FIG. 3 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments.

[0025] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples and the present invention is not limited thereto.

[0026] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing embodiments of the present invention and should not be limited in any way. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0027] FIG. 1 is a block diagram illustrating a text analysis and speech synthesis device (100) according to one embodiment of the present invention. The disclosed text analysis and speech synthesis device (100) is configured to analyze and separate a given text into utterance units, and synthesize speech in chunk units based on the analysis and separation. As illustrated, the text analysis and speech synthesis device (100) according to one embodiment includes a text analysis module (102), a text encoding module (104), and a speech synthesis module (106).

[0028] The text analysis module (102) separates the input text into utterance units and generates multiple text chunks.

[0029] The text encoding module (104) encodes a plurality of text chunks generated by the text analysis module (102) to generate a plurality of text feature chunks.

[0030] The speech synthesis module (106) generates a speech signal based on a plurality of text feature chunks.

[0031] Below, a text analysis and voice synthesis device (100) according to one embodiment of the present invention having the above configuration will be described in more detail.

[0032] Analysis of speech units in text

[0033] The text analysis module (102) divides the input text into utterance units and generates multiple text chunks. In the disclosed embodiments, an utterance unit refers to a unit that a person reads while speaking a given text.

[0034] In one embodiment, the text analysis module (102) may segment the input text into utterance units based on one or more of the following: a sentence-ending character, a final word ending, a conjunction, a connecting word ending, or the boundary of one or more phrases or clauses detected in the input text. This is described in more detail as follows.

[0035] In one embodiment, the text analysis module (102) may divide the input text into sentence units to generate one or more text chunks. To this end, the text analysis module (102) may detect sentence boundaries by searching for sentence-ending characters or by utilizing morphological analysis or part-of-speech tagging to search for final word endings.

[0036] In another embodiment, the text analysis module (102) may segment the input text based on its connection relationships to generate one or more text chunks. For example, if the text analysis module (102) cannot detect sentence boundaries within the input text, the text analysis module (102) may segment the input text based on its equivalence relationship, dependency relationship, etc. Additionally, the text analysis module (102) may utilize morphological analysis or part-of-speech tagging to search for conjunctions or connecting suffixes and segment the input text using these.

[0037] In another embodiment, the text analysis module (102) can predict a person's typical breathing patterns to segment text and generate one or more text chunks. People naturally speak with pauses based on the boundaries of phrases and clauses in sentences. Based on this typical human speech pattern, the text analysis module (102) can identify the locations of phrases and clauses in a given text through phrase structure analysis or dependent syntax analysis, and segment the input text based on these boundaries.

[0038] In addition to the methods described above, the text analysis module (102) can separate input text into utterance units using various methods. In one embodiment, the text analysis module (102) can separate input text by selecting or combining one or more of the above-described methods, taking into account the characteristics of the input text and the purpose of speech synthesis. The operating conditions and order of each method may be changed as needed.

[0039] In one embodiment, the text analysis module (102) may, before or after separating the input text, delete special characters unnecessary for speech from the input text or replace them with other characters. In addition, in one embodiment, the text analysis module (102) may, before or after separating the input text, replace numbers, foreign words, foreign languages, etc. included in the input text with their corresponding phonetic equivalents. In addition, in one embodiment, the text analysis module (102) may, before or after separating the input text, convert at least a portion of the input text into phonemes as they are pronounced to ensure naturalness of the synthesized voice.

[0040] In one embodiment, if the text analysis module (102) determines that the length of the input text is less than a preset reference value or that the input text cannot be divided into utterance units or that it is inappropriate to divide the input text, the text analysis module (102) may pass the input text to the text encoding module (104) as is without dividing the input text into utterance units.

[0041] Text encoding

[0042] The text encoding module (104) encodes a plurality of text chunks generated by the text analysis module (102) to generate a plurality of text feature chunks. In one embodiment, the text encoding module (104) may be configured to generate text feature chunks required for speech synthesis based on contextual information of input text separated into utterance units. In one embodiment, the contextual information of the input text may be obtained using various text analysis or text mining algorithms.

[0043] In one embodiment, the text encoding module (104) may use one or more artificial neural network structures, such as a convolutional neural network, a recurrent neural network, and a transformer, or a combination thereof, to generate a plurality of text feature chunks to encode pronunciation information or contextual information of the input text.

[0044] In the disclosed embodiments, the text encoding module (104) may encode text chunks sequentially or in batches to optimize the response time or performance speed of the encoder of the text encoding module (104).

[0045] For example, in one embodiment, the text encoding module (104) can sequentially encode a plurality of text chunks received from the text analysis module (102).

[0046] In another embodiment, the text encoding module (104) may organize the plurality of text chunks received from the text analysis module (102) into one or more batches by considering the length of the plurality of text chunks, and perform encoding for each batch.

[0047] For example, the text encoding module (104) may batch text chunks of similar length together in order to encode text chunks of variable length at once. Alternatively, the text encoding module (104) may sort the text chunks received from the text analysis module (102) by length and sequentially group the sorted text chunks to form a batch. In some embodiments, the above-described rearrangement and grouping of text chunks may be performed by the text analysis module (102) rather than the text encoding module (104). In this case, the text encoding module (104) may be configured to encode the text chunks received from the text analysis module (102) in batch units. When the text chunks are rearranged or grouped in this way to form a batch, the text encoding speed can be optimized. After encoding the text chunks, the text encoding module (104) may restore the order of the encoded text feature chunks to match the original input text and provide the restored order to the speech synthesis module (106).

[0048] In one embodiment, the text encoding module (104) may perform encoding of a specific text chunk using a previous text chunk of the specific text chunk or information related to the previous text chunk (e.g., text feature chunk or state information for the previous text chunk, etc.) to improve the quality of the synthesized speech.

[0049] Additionally, in one embodiment, the text encoding module (104) may encode a plurality of text chunks using context prediction information for each of the plurality of text chunks to improve the quality of the synthesized speech.

[0050] In one embodiment, the text encoding module (104) may cache encoded text feature chunks to optimize response time or performance speed during the encoding process. For example, the text encoding module (104) may cache the text feature chunks in a cache memory provided on a CPU, GPU, or hard disk.

[0051] speech synthesis

[0052] The speech synthesis module (106) generates a speech signal based on a plurality of text feature chunks. In the disclosed embodiment, the speech synthesis module (106) is configured to synthesize speech in chunk units based on encoded utterance unit text features. Chunk-based speech synthesis refers to a method of synthesizing speech for a given text sequentially by dividing it into small time-unit speech signals rather than synthesizing it all at once.

[0053] In one embodiment, the speech synthesis module (106) is configured to decode a plurality of text feature chunks to generate a plurality of speech chunks, and to concatenate the generated plurality of speech chunks to generate a speech signal. The speech synthesis module (106) may synthesize speech by using one or more artificial neural network structures, such as a convolutional neural network, a recurrent neural network, and a transformer, or a combination thereof, to generate speech chunks based on encoded pronunciation information or contextual information. At this time, the speech synthesis module (106) may synthesize speech chunks in a fixed-length or variable-length manner. In addition, the length of the synthesized speech chunk may not necessarily match the length of the text chunk. That is, the length of the speech chunk may be different from the length of the utterance unit text chunk.

[0054] In the disclosed embodiments, the speech synthesis module (106) may decode text feature chunks sequentially or in batches to optimize the decoding performance speed.

[0055] For example, in one embodiment, the speech synthesis module (106) can sequentially decode a plurality of text feature chunks received from the text encoding module (104) to generate a speech chunk.

[0056] In another embodiment, the speech synthesis module (106) may generate speech chunks by composing the plurality of text feature chunks into one or more batches, taking into account the lengths of the plurality of text feature chunks received from the text encoding module (104), and performing decoding for each batch.

[0057] For example, the speech synthesis module (106) may batch text feature chunks of similar lengths together in order to decode text feature chunks of variable lengths at once. Alternatively, the speech synthesis module (106) may sort the text feature chunks received from the text encoding module (104) by length and sequentially group the sorted text feature chunks to form batches. In some embodiments, the rearrangement and grouping of text feature chunks as described above may be performed by the text encoding module (104) rather than the speech synthesis module (106). In this case, the speech synthesis module (106) may be configured to encode the text feature chunks received from the text encoding module (104) in batch units. When batches are formed by rearranging or grouping text feature chunks in this way, the decoding speed of the text features can be optimized. The speech synthesis module (106) can generate a synthetic speech by restoring the order of the generated speech chunks to match the original input text after decoding the text feature chunks.

[0058] In one embodiment, the speech synthesis module (106) may cache decoded speech chunks to optimize the response time or performance speed of speech synthesis. For example, the speech synthesis module (106) may cache the speech chunks in cache memory provided in a CPU, GPU, or hard disk.

[0059] In one embodiment, the text encoding module (104) may, if the length of a specific text feature chunk among the generated plurality of text feature chunks is greater than or equal to a threshold value, re-divide the specific text feature chunk into one or more sub-feature chunks and provide the same to the speech synthesis module (106). In this case, the speech synthesis module (106) may decode each of the received one or more sub-feature chunks to generate one or more sub-voice chunks, and may connect the generated one or more sub-voice chunks to generate a speech chunk corresponding to the specific text feature chunk. In this way, when a text feature chunk exceeding a certain length is re-divided into sub-feature chunks and provided to the speech synthesis module (106), the start time of speech synthesis can be reduced compared to providing the entire text feature chunk to the speech synthesis module (106).

[0060] In one embodiment, the speech synthesis module (106) may concatenate text features to improve the quality of the synthesized speech. For example, the speech synthesis module (106) may concatenate one or more text feature chunks among a plurality of text feature chunks and decode the concatenated text feature chunks to generate a speech signal.

[0061] In one embodiment, a separate attention module or the like may be used to align the reference time between the text encoding module (104) and the speech synthesis module (106). In addition, a data structure such as a queue may be used to implement this.

[0062] In one embodiment, the speech synthesis module (106) may store the speech synthesis state of a previous text chunk and use it in synthesizing the next text chunk to enhance the expressiveness of the speech. That is, the speech synthesis module (106) may decode a specific text feature chunk using information from one or more previous text feature chunks for a specific text feature chunk among a plurality of text feature chunks. At this time, a separate artificial neural network module or the like may be used to store the speech synthesis state.

[0063] In one embodiment, the speech synthesis module (106) may generate a speech signal by additionally considering one or more of the emotional information and speech style information corresponding to the input text. For example, to enhance the expressiveness of speech, the speech synthesis module (106) may additionally receive characteristics such as emotional and style information in addition to the text input, and utilize these to generate the speech signal.

[0064] In one embodiment, the speech synthesis module (106) may provide some of the synthesis results of the speech chunks to the client or another server en route to increase response time.

[0065] FIG. 2 is a flowchart illustrating a text analysis and speech synthesis method (200) according to one embodiment. The illustrated method may be performed in a computing device having one or more processors and a memory storing one or more programs executed by the one or more processors, for example, an effective neural network architecture generation device (100) according to one embodiment. In the illustrated flowchart, the method or process is described by dividing it into a plurality of steps, but at least some of the steps may be performed in a changed order, combined with other steps and performed together, omitted, divided into detailed steps and performed, or one or more steps not illustrated may be added and performed.

[0066] In step 202, the text analysis module (102) separates the input text into utterance units to generate multiple text chunks.

[0067] In step 204, the text encoding module (104) encodes the generated plurality of text chunks to generate a plurality of text feature chunks.

[0068] In step 206, the speech synthesis module (106) generates a speech signal based on a plurality of text feature chunks.

[0069] According to the disclosed embodiments, input text is divided into utterance units, which are units read by humans, and each utterance is synthesized into speech, thereby eliminating awkwardness at the boundaries between text chunks. Furthermore, rather than encoding the entire input text and then synthesizing speech based on it, encoding and speech synthesis are performed on a per-text chunk basis, thereby optimizing the time required until speech synthesis begins.

[0070] For example, conventional speech synthesis techniques require encoding the entire context of the input text to achieve natural-sounding speech synthesis. Therefore, encoding the entire input text is performed first, text features are generated for the input text, and then speech is decoded using the generated text features. However, this method has the drawback that speech synthesis can only be performed after encoding the entire input text is complete. This is a major cause of response time delay, especially when synthesizing text into speech in real time.

[0071] To address this, the disclosed embodiments first divide the text into utterance units, and then perform text feature encoding and speech synthesis for each divided utterance unit. In this case, speech synthesis can be performed immediately after encoding the first text chunk, rather than encoding the entire text, thereby effectively reducing the time required for speech synthesis to begin compared to conventional methods. In addition, by dividing the text into utterance units, which are units that humans read in chunks, rather than arbitrarily dividing the text, the encoding can be performed on a text chunk basis without considering the contextual information of the entire text, while minimizing awkwardness that may occur at boundaries of the synthesized speech, etc.

[0072] Figure 3 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

[0073] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be the aforementioned text analysis and speech synthesis device (100).

[0074] A computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may cause the computing device (12) to operate according to the exemplary embodiments mentioned above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, which, when executed by the processor (14), may be configured to cause the computing device (12) to perform operations according to the exemplary embodiments.

[0075] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data, and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by the processor (14). In one embodiment, the computer-readable storage medium (16) may be a memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, any other form of storage medium that can be accessed by the computing device (12) and store desired information, or a suitable combination thereof.

[0076] A communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and computer-readable storage media (16).

[0077] The computing device (12) may also include one or more input / output interfaces (22) that provide interfaces for one or more input / output devices (24) and one or more network communication interfaces (26). The input / output interfaces (22) and the network communication interfaces (26) are connected to the communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) via the input / output interfaces (22). Exemplary input / output devices (24) may include input devices such as pointing devices (such as a mouse or a trackpad), a keyboard, a touch input device (such as a touchpad or a touchscreen), a voice or sound input device, various types of sensor devices and / or photographing devices, and / or output devices such as display devices, printers, speakers and / or network cards. The exemplary input / output devices (24) may be included within the computing device (12) as a component constituting the computing device (12), or may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0078] While the present invention has been described in detail above through representative examples, those skilled in the art will understand that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims set forth below but also by equivalents thereof.

[0079] <Explanation of symbols>

[0080] 100: Text Analysis and Speech Synthesis Device

[0081] 102: Text Analysis Module

[0082] 104: Text Encoding Module

[0083] 106: Speech Synthesis Module

Claims

1. A text analysis module that separates input text into utterance units and generates multiple text chunks; A text encoding module that encodes the generated plurality of text chunks to generate a plurality of text feature chunks; and A text analysis and speech synthesis device comprising a speech synthesis module that generates a speech signal based on the plurality of text feature chunks.

2. In claim 1, A text analysis and speech synthesis device, wherein the text analysis module divides the input text into utterance units based on one or more of a sentence ending character, a final word ending, a conjunction, a connecting word ending included in the input text, or a boundary of one or more phrases or clauses searched for in the input text.

3. In claim 1, A text analysis and speech synthesis device, wherein the text encoding module generates the plurality of text feature chunks using one or more artificial neural network structures selected from the group consisting of a convolutional neural network, a recurrent neural network, and a transformer.

4. In claim 1, A text analysis and speech synthesis device, wherein the text encoding module sequentially encodes the plurality of text chunks received from the text analysis module.

5. In claim 1, A text analysis and speech synthesis device in which the text encoding module considers the length of the plurality of text chunks received from the text analysis module and organizes the plurality of text chunks into one or more batches, and performs encoding for each batch.

6. In claim 1, A text analysis and speech synthesis device, wherein the text encoding module encodes a specific text chunk among the plurality of text chunks by using information of one or more previous text chunks for the specific text chunk.

7. In claim 1, A text analysis and speech synthesis device, wherein the text encoding module encodes the plurality of text chunks using context prediction information for each of the plurality of text chunks.

8. In claim 1, The above speech synthesis module is a text analysis and speech synthesis device that decodes the plurality of text feature chunks to generate a plurality of speech chunks and connects the generated plurality of speech chunks to generate the speech signal.

9. In claim 1, The text encoding module is a text analysis and speech synthesis device, wherein, if the length of a specific text feature chunk among the plurality of generated text feature chunks is greater than a threshold value, the specific text feature chunk is re-divided into one or more sub-feature chunks and provided to the speech synthesis module.

10. In claim 9, A text analysis and speech synthesis device, wherein the speech synthesis module decodes each of the one or more sub-feature chunks to generate one or more sub-voice chunks, and connects the generated one or more sub-voice chunks to generate a speech chunk corresponding to the specific text feature chunk.

11. In claim 1, A text analysis and speech synthesis device, wherein the speech synthesis module concatenates one or more text feature chunks among the plurality of text feature chunks and decodes the concatenated text feature chunks to generate the speech signal.

12. In claim 1, A text analysis and speech synthesis device, wherein the speech synthesis module sequentially decodes the plurality of text feature chunks received from the text encoding module to generate the speech signal.

13. In claim 1, A text analysis and speech synthesis device in which the speech synthesis module organizes the plurality of text feature chunks received from the text encoding module into one or more batches by considering the length of the plurality of text feature chunks, and performs decoding for each batch to generate the speech signal.

14. In claim 1, A text analysis and speech synthesis device, wherein the speech synthesis module decodes a specific text feature chunk by using information of one or more previous text feature chunks for the specific text feature chunk among the plurality of text feature chunks.

15. In claim 1, The above-mentioned speech synthesis module is a text analysis and speech synthesis device that generates the speech signal by additionally considering at least one of emotional information and speech style information corresponding to the input text.

16. A method performed on a computer, A step of generating multiple text chunks by dividing the input text into utterance units; A step of encoding the generated plurality of text chunks to generate a plurality of text feature chunks; and A text analysis and speech synthesis method, comprising the step of generating a speech signal based on the plurality of text feature chunks.

Images