High-speed speech dialogue system using parallel execution pipeline for response text generation via generative sequence segmenting into phrase segments and speech synthesis

Abstract

To provide a speech dialogue system that utilizes machine learning models including speech recognition models, speech synthesis models, and language models, enhances pipeline parallelism for texts with rather complex sentence structures, improves overall throughput, and reduces a response time until a user perceives a speech signal.SOLUTION: In a speech dialogue system, a text processing unit for monitoring tokens sequentially generated by a text generation unit with language models by using machine learning models including speech recognition models, speech synthesis models, and the language models, divides a generated text in response to detecting a phrase marker and contents of the marker, immediately before or after the marker, initiates execution of speech synthesis without awaiting the end of generation of the entire response text or one sentence, and executes in parallel remaining text generation and speech synthesis processing in the sentence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a spoken dialogue system that uses machine learning models including a speech recognition model, a speech synthesis model, and a language model. [Background technology]

[0002] Various methods for improving the response speed of voice dialogue systems have been studied. The motivation is to maximize the accuracy of speech recognition, the quality of the response content, and the quality of the synthesized speech, while shortening the latency to achieve a more human-like reaction time. Generally, when the time it takes for the other party's response to arrive after the end of an utterance exceeds one second, humans feel that they are not speaking to a human and experience stress.

[0003] There was generally a trade-off between the quality of the response content and the computational time required to output the response content. Before large-scale language models became practical, response content was mainly generated by pattern matching using a machine learning model between a huge response content dictionary database and the input text. For example, a machine learning model was used in which a huge number of possible response sentences were stored in a database, the input text converted into a feature vector was used as an input value, and the index of the most likely response sentence was output. Non-Patent Document 1 describes a machine learning encoder that performs pre-learning using a search-based response selection task to efficiently and quickly select response sentences, and assumes the existence of a response sentence database. On the other hand, the practical application of large-scale language models that have been pre-trained on vast text sequences has enabled the model to generate appropriate responses even without a response sentence database. Large-scale language models generate one token per inference step. This inference step corresponds to a clock cycle in computer architecture. One token roughly corresponds to one word or subword. Special tokens such as an end token, which indicates the completion of generation, are also output. When a language model is incorporated into a pipeline of a spoken dialogue system, the simplest approach is to execute the generation process until the clock cycle reaches an end token, and then transfer the generated text to the next pipeline based on the detection of the end token. In this case, the process from response text generation to speech synthesis is linear, equivalent to a scalar processor with a single execution unit in computer architecture. This design imposes a speed constraint on the long latency until the response content is generated, and also creates a waiting state for computational resources, particularly multiple GPU processes. Alternatively, there are designs that use the detection of a period or punctuation mark at the end of a sentence as a key for transferring the text to the speech synthesis pipeline. Even in this case, if a sentence is long, even if the language model's clock cycle generates 1,000 tokens per second, it takes 0.2 seconds to generate 20 tokens, and when the latency of the other pipelines is added together, it easily exceeds 1.0 second. To fundamentally solve this problem, a method has been considered to integrate the text generation language model and the speech synthesis model and train them as a single model, which would significantly reduce latency but would also lose the independence of each module. Non-Patent Document 2 proposes a generation model that takes speech and auxiliary text as input values ​​and can directly output a response speech without going through a text generation unit. By integrating the text generation unit and speech synthesis unit, it is possible to eliminate the latency required from response sentence generation to speech synthesis, but as mentioned above, there is no independence of the modules, and it has the architectural drawback of low customizability. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Matthew Henderson, Inigo Casanueva, Nikola Mrksic, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulic.2020.ConveRT:Efficient and Accurate Conversational Representations from Transformers.In Findings of the Association for Computational Linguistics:EMNLP 2020,pages 2161-2174,Online.Association for Computational Linguistics.” [Non-patent document 2] “Le, Matthew, et al. “Voicebox:Text-guided multilingual universal speech generation at scale.”Advances in neural information processing systems 36(2024).” Summary of the Invention [Problem to be solved by the invention]

[0005] Spoken dialogue systems that use large-scale language models for response generation enable more flexible, human-like judgments and diverse inferences than existing systems based on response selection tasks. However, they require a different architecture than existing spoken dialogue system pipelines, and have been underdeveloped. In particular, response speed is an issue. For example, a response selection task-based dialogue system can determine a response in a single inference run, whereas a large-scale language model performs inference on a token-by-token basis. This results in delays due to the sequential processing cycles required to generate an average of 30 to 40 words that make up a sentence. However, since the response speed of human-to-human spoken dialogue is typically less than one second, reducing the delay to less than one second is essential for a comfortable spoken dialogue experience. This requires not only high-performance computing resources but also ingenuity in the system architecture, but no established methodology exists. [Means for solving the problem]

[0006] The voice dialogue system of the present invention, which operates on a computer terminal, comprises: Of the processing steps, which consist of a speech input section, speech input buffer, speech recognition section, text generation section using a language model, text processing section, speech synthesis section, and speech output section, the pipeline from the text generation section to the speech synthesis section is highly parallelized to improve throughput. The text generation section detects phrase boundaries as markers from the tokens sequentially output by the text generation section, divides the generated text just before or just after the marker depending on the content of the marker, and transfers the text to the speech synthesis section before the entire sentence is generated. This reduces the computational resources waiting for execution and improves parallelism. This shortens the time until the first speech signal reaches the user, reducing delays and speeding up speech responses while maintaining natural speech synthesis quality. [Effects of the Invention]

[0007] In this study, we improve the throughput from response generation to speech synthesis, thereby accelerating response speed. These include the main factors that contribute to delays in voice dialogue systems that primarily use machine learning models: speech input, end-of-utterance detection (EOU), speech recognition, response generation using a language model, text processing, speech synthesis, speech output, and overhead associated with data transfer, compression encoding, and decoding. Assuming a language model clock cycle of 1,000 tokens / second, an average sentence count of 35 words, an average sentence count of 2 phrases, an average corresponding word count per token of 1 word, and a marker detection time of ε<0.000001 seconds, the average duration from the start of response generation to the start of synthesis of the first speech signal is reduced from 0.35 seconds to 0.175 seconds. Assuming the average duration from the end of speech input utterance to the start of playback of the response speech signal is 0.9 seconds, the reduced time accounts for approximately 19.4% of the overall delay pipeline, resulting in an average improvement in response speed of approximately 20%. [Brief explanation of the drawings]

[0008] [Figure 1] This is a conceptual diagram illustrating the entire pipeline that constitutes the spoken dialogue system of the present invention as set forth in claim 1. This pipeline is a typical pipeline that constitutes a spoken dialogue system that uses a machine learning model. [Figure 2a] 1 illustrates a processing pipeline according to a general architecture in which a language model is used as an utterance content sentence generator and the method of the present invention is not used. [Figure 2b] 1 illustrates a processing pipeline based on a parallel architecture when the method of the present invention is used, with a language model as the utterance content sentence generator. [Figure 3] The figure shows two patterns in which the text processor identifies markers that separate phrases and divides the generated text, namely, dividing immediately before and immediately after the marker. DETAILED DESCRIPTION OF THE INVENTION

[0009] The voice dialogue system of the present invention operates on a computer terminal, and as long as it includes each of the aforementioned units from the voice input unit to the voice output unit, any of these units may be distributed across different terminals. For example, in a voice dialogue system that is implemented only within a local terminal without going via the Internet, each unit is located on a single computer terminal. On the other hand, in a cloud-based voice dialogue system, only the voice input unit and voice output unit are located on the local terminal, and the rest are located on a server terminal connected via the Internet, or the voice recognition unit and voice synthesis unit may be located on the local terminal. The combination is arbitrary, and any combination is acceptable for the present invention.

[0010] The role of each component constituting the voice dialogue system envisioned by the present invention will be described below. The voice input unit generally converts physical vibrations in the air into electrical signals via a microphone, and converts the analog signals into discrete representations such as PCM if the computer terminal is digital, and then captures the voice information by storing it in the computer's memory space in binary, WAV, mp3, etc. The voice input unit can be anything that can physically record spoken voice signals in memory space. The voice input buffer is a memory area on the computer terminal that holds continuously input voice signals until they are transferred to the next processing unit, the voice recognition unit. The speech recognition unit receives the speech signal recorded by the speech input unit and transferred from the speech input buffer, and outputs the natural language expression that is estimated to be spoken from the speech signal as text data. The output text data can be encoded in any way, such as Unicode, ASCII string, or a sequence of subword token indices. The language model-based text generation unit accepts inputs including text data output from the speech recognition unit and, if necessary, any additional information, and outputs the response content corresponding to the input or the additional information as necessary as a text sequence, i.e., a string of tokens or information obtained by encoding words in any format, usually one token or one encoding unit at a time for each inference step. As explained for the speech recognition unit, the format of the output text data can be encoded in any way. Furthermore, the information accompanying the response content can be in any form. For example, the response can be output accompanied by an image or the like. The text processing unit analyzes and, if necessary, manipulates and modifies the code strings sequentially generated and output from the text generating unit. The speech synthesis unit inputs the text data generated by the language model as the response content and outputs a speech signal as a read-out voice. The data format of the output speech signal can be any format, such as binary, WAV, or encoded in any compression format. The audio output unit converts the digital data of the audio signal output by the audio synthesis unit into physical vibrations that can be perceived by the human ear, and outputs the vibrations through any device such as a speaker or headphones.

[0011] The speech recognition model and language model mentioned above can be separate models or integrated. For example, if they are all integrated, the model inputs voice data and generates a response directly from the input voice without first transcribing it and outputting it. The language model and speech synthesis model must each be independent models.

[0012] The models that realize the speech recognition model, language model, and speech synthesis model may be trained using machine learning algorithms or realized by any other means, as long as they provide the desired input and output functions described above.

[0013] Voice signals input through the voice input unit are usually stored sequentially in a voice input buffer, and then transferred to the voice recognition unit, which is the next processing pipeline, when certain conditions are met. Similarly, the audio signal transferred to the audio output unit is temporarily stored in the audio output buffer, and when certain conditions are met, it is reproduced in the audio output unit, which is the next processing pipeline. These buffers may be omitted as long as they fulfill the purposes of the aforementioned voice input / output, voice recognition, and voice synthesis sections. The timing at which the audio output unit plays back an audio signal can be controlled within a certain range by the program, and in particular, the start of playback can be controlled at a certain point between the arrival of an audio signal and the arrival of the next signal.

[0014] The language processing unit may use any method to identify phrases. Furthermore, the definition of a "phrase" may be any as long as the division is natural enough for humans to perceive it as such. For example, in English, conjunctions such as "and" and "when" may be used as markers for identification, and in Japanese, conjunctions such as "or" and "because" may be used as markers for identification. Even when morpheme information is used, any method for morpheme analysis may be used.

[0015] The language processor may divide the generated text in any way depending on the content of the marker. For example, if the marker is a conjunction such as "and," "or," or "because," the text may be divided before the marker, and if the marker is a comma (,) or "for example," the text may be divided after the marker.

[0016] A more specific embodiment will now be described. In particular, the details of how the sequence generated by the text generation unit consisting of a language model is divided in the text processing unit, and how speech synthesis processing and text generation processing are executed in parallel on a pipeline will be explained. For example, a speech signal representing the user's utterance, "What is the origin of all things?", is stored in a speech input buffer and transferred to the speech recognition unit. The transcribed text is then input into the language model of the text generation unit, encoded as input tokens. The language model attempts to generate the most likely response to the input sequence: "There is no answer to that question yet, but there are generally two models that explain the origin of all things: the ceramic model and the fully automated model. The former requires a creator, and all things were created from the synthesis of reduced elements. The latter, on the other hand, assumes that the current world was created by autonomous movement from chaos." In the generation process, one token is generated sequentially at each computational step; the entire text is not output in a single inference. Therefore, for example, "that" is output first, then "question" is generated in the next step, and then "to" is generated in the next step, and so on. The text processing unit monitors the tokens generated at each step, and when ", but" is generated after a certain number of steps, it is used as a phrase marker to segment the sentence. Assume that when "but" is detected, it is determined that it should be split immediately before the "but." The text processor immediately transfers the text generated and stored up to "but," "There is no answer to that question yet," to the speech synthesis unit. Upon receiving this, the speech synthesis unit begins speech synthesis. While "There is no answer to that question yet," is synthesized into a speech signal in a separate execution process from the text generation process, the text generation process is executing in parallel the generation of the text following "but," "Generally, as a model to explain the origin of all things...." A certain number of steps later, the text processor detects ", and" as a marker and transfers the text generated up to that point, "Generally, as a model to explain the origin of all things, one is the ceramic model," to the speech synthesis unit, which then begins synthesizing it. However, here, it is determined that "and," like "but," should be split immediately before the "but."Before the synthesis of the following text, "But generally...", the speech synthesis process for "There is no answer to that question yet," is completed, and the synthesized speech signal is transferred directly to the speech output unit, where it has already started to be played and perceived by the user. While the user is listening to the speech signal for "The answer to that question is...", the text after ", but" is being processed.

[0000] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims.

Claims

1. In a speech dialogue system that runs on a computer terminal, It has a voice input unit, a voice input buffer, a voice recognition unit, a text generation unit using a language model, a text processing unit, a voice synthesis unit, and a voice output unit, A voice dialogue system in which voice input from the voice input unit is stored in the voice input buffer and then converted into text by the voice recognition unit, response text to the input text is sequentially generated as output tokens by the text generation unit, the text processing unit analyzes and edits the generated content and then transfers it to the voice synthesis unit, the voice synthesis unit converts the response text transferred from the text processing unit into a voice signal and transfers it to the voice output unit, and then the voice output unit outputs the voice signal as a physical sound wave, The text processing unit detects markers that separate phrases from the response text sequence that the text generation unit sequentially generates as output tokens, and when a marker is detected, divides the sequence at a position immediately before or after the marker according to the word or morpheme of the marker, and then transfers the divided elements in time series from the previous element to the speech synthesis unit, and speech synthesis processing is started before the text generation unit finishes generating an entire sentence, Response text generation and speech signal generation are performed in parallel without waiting for the completion of generation of the entire sentence by the text generation section, thereby shortening the overall pipeline, improving throughput, and accelerating response speed while maintaining natural speech output. A voice dialogue system characterized by:

2. The speech dialogue system according to claim 1 A voice dialogue system that enables high-speed responses even in a web browser by executing the voice input unit, voice input buffer, and voice output unit described in claim 1 using a computer program that runs on a web browser, and by executing the other units described in claim 1 on a computer terminal or process that is capable of communicating with and different from the process on which the web browser runs.

Images