Speech dialogue system eliminating need for end-of-utterance detection, website-embedded voice dialogue q&a system

Abstract

To provide a speech dialogue system and speech dialogue Q&A system where delays associated with end-of-utterance (EOU) detection processing are eliminated, significantly improving overall response speed.SOLUTION: An end-of-utterance detection unit is removed from a general speech dialogue system having a speech input unit, a speech input buffer, an end-of-utterance detection unit, a speech recognition unit, a text generation unit using a language model, a speech synthesis unit, and a speech output unit, and, instead, a trigger button that actively indicates that a speech signal is being input is added. While the trigger button is pressed, the input speech signal is held in the speech input buffer, and when the button is released, the speech signal is transferred to the speech recognition unit.SELECTED DRAWING: Figure 1b

Description

[Technical Field]

[0001] The present invention relates to a spoken dialogue system that uses machine learning models including a speech recognition model, a speech synthesis model, and a language model. [Background technology]

[0002] Many efforts have been made to improve the pipeline of spoken dialogue systems, with the motivation being to combine high-accuracy speech recognition with complex response text generation and high-quality synthesized speech, while reducing response latency and improving real-time performance.

[0003] For example, in a voice dialogue system built into a home speaker product, the speaker inputs speech to the system while moving freely around the indoor space, so the system needs to detect the end of the utterance using information from the speech input itself. End-of-utterance detection (EOU) processing originally started with a method of counting silent intervals and determining the end of an utterance when their duration exceeded a threshold, but more recently, there are methods to improve detection performance by training a machine learning model that inputs phonetic information centered on the fundamental frequency F0 and the content of the utterance obtained from speech recognition. Non-Patent Document 1 compares ASR hypothesis embedding and acoustic embedding and shows that the phonetic information provided by acoustic embedding contributes most to the model performance. In Non-Patent Document 2, a multimodal model using speech and text is trained to build turn-taking capabilities that determine when to end an utterance or when an interruption is possible, particularly through a user state detection function. However, no matter what method is used, a certain delay is unavoidable between the actual end of an utterance and the detection of its end. Even cutting-edge methods using acoustic information require a delay of around 0.5 seconds to detect the end of an utterance. This occupies a large portion of the overall time required for a spoken dialogue system consisting of a multi-layered pipeline, and is a limiting factor that impairs real-time performance. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Maas, Roland, et al. “Combining acoustic embeddings and decoding features for end-of-utterance detection in real-time far-field speech recognition systems.”2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018” [Non-patent document 2] “Lin, Ting-En, et al. “Duplex conversation: Towards human-like interaction in spoken dialogue systems.” Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.2022.” Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, to improve the response speed of voice dialogue systems such as Amazon Alexa and Google Home, end-of-utterance detection (EOU) using machine learning models that use prosodic information as input has been used, but this unit alone inevitably increases delay by around 0.5 seconds, resulting in a latency of over one second from voice input to response.Since the response speed in voice dialogue between humans is usually less than one second, a dialogue system that consistently takes more than one second to respond does not provide the same comfort as a human-to-human voice dialogue experience, and inevitably causes unconscious stress and resistance. Therefore, in order for interactive systems that use voice input and output as an interface to be fully implemented in society, it was necessary to solve this problem of response speed. [Means for solving the problem]

[0006] The voice dialogue system of the present invention, which operates on a computer terminal, comprises: In addition to the speech input unit, speech input buffer, speech recognition unit, text generation unit using a language model, speech synthesis unit, and speech output unit, there is also a trigger button that is pressed while a speech signal is being input. While this trigger button is pressed, the speech input signal is held in the speech input buffer, and when the trigger button is released, the speech signal held up to that point is sent to the speech recognition unit, and subsequent processing is executed. This eliminates the delay associated with end-of-utterance (EOU) processing, and the overall response speed is the sum of delays from the speech recognition unit, language model, speech synthesis unit, speech output unit, and overhead associated with data transfer, compression encoding, decoding, etc. [Effects of the Invention]

[0007] In this invention, the overall delay of a voice dialogue system that mainly uses a machine learning model is made zero by the pipeline of speech input, end-of-utterance detection (EOU), speech recognition, text generation using a language model, speech synthesis, speech output, and overhead associated with data transfer, compression encoding, and decoding. End-of-utterance detection can account for around 30% to 50% of this time, and reducing this can significantly reduce overall delay and improve response speed. For example, even when text generation processing is performed using a large-scale language model with 70 billion parameters, by optimizing each processing unit, combining the networking that connects each unit with parallelization of each processing unit, it is possible to reduce overall delay to within 0.5 seconds, thereby bringing the real-time performance of a voice dialogue system to the same level of delay as human conversation. [Brief explanation of the drawings]

[0008] [Figure 1a] FIG. 1 is a conceptual diagram illustrating a pipeline of a conventional general spoken dialogue system that mainly uses a machine learning model, including a speech end detection process. [Figure 1b] 1 is a conceptual diagram illustrating the entire pipeline constituting the voice dialogue system of the present invention as set forth in claim 1.

[0023] FIG. [Figure 2] FIG. 10 is a conceptual diagram illustrating the pipeline of the entire website-embedded voice dialogue Q&A system, in which the trigger button is realized as an HTML element on the web browser, as described in claim 2. DETAILED DESCRIPTION OF THE INVENTION

[0009] The voice dialogue system of the present invention operates on a computer terminal, and as long as it includes each of the aforementioned units from the voice input unit to the voice output unit, any of these units may be distributed across different terminals. For example, in a voice dialogue system that is implemented only within a local terminal without going via the Internet, each unit is located on a single computer terminal. On the other hand, in a cloud-based voice dialogue system, only the voice input unit and voice output unit are located on the local terminal, and the rest are located on a server terminal connected via the Internet, or the voice recognition unit and voice synthesis unit may be located on the local terminal. The combination is arbitrary, and any combination is acceptable for the present invention.

[0010] The role of each component constituting the voice dialogue system envisioned by the present invention will be described below. The voice input unit generally converts physical vibrations in the air into electrical signals via a microphone, and converts the analog signals into discrete representations such as PCM if the computer terminal is digital, and then captures the voice information by storing it in the computer's memory space in binary, WAV, mp3, etc. The voice input unit can be anything that can physically record spoken voice signals in memory space. The voice input buffer is a memory area on the computer terminal that holds continuously input voice signals until they are transferred to the next processing unit, the voice recognition unit. The speech recognition unit receives the speech signal recorded by the speech input unit and transferred from the speech input buffer, and outputs the natural language expression that is estimated to be spoken from the speech signal as text data. The output text data can be encoded in any way, such as Unicode, ASCII string, or a sequence of subword token indices. The language model-based text generation unit receives input containing text data output from the speech recognition unit and, if necessary, any additional information, and outputs a response corresponding to the input or additional information as necessary as text data. As with the speech recognition unit, the format of the output text data can be encoded in any way. Furthermore, the information accompanying the response can be in any form. For example, an image or the like can be output. The speech synthesis unit inputs the text data generated by the language model as the response content and outputs a speech signal as a read-out voice. The data format of the output speech signal can be any format, such as binary, WAV, or encoded in any compression format. The audio output unit converts the digital data of the audio signal output by the audio synthesis unit into physical vibrations that can be perceived by the human ear, and outputs the vibrations through any device such as a speaker or headphones.

[0011] The aforementioned speech recognition model, language model, and speech synthesis model can be separate models, or they can be an integrated model. For example, if all of these are integrated, the model inputs speech data, generates a response content within the model, and then directly outputs the speech data of the response content. Alternatively, if only the language model and speech synthesis model are integrated, text information is input to the model and response speech data is output. If only the speech recognition model and language model are integrated, speech data is input and response text is output.

[0012] In the dialogue system of the present invention, the speech synthesis unit and speech output unit may be omitted as long as the response content from the system is sufficient for establishing a dialogue. For example, if the response text generated by the language model is visually presented to the speaker as text information and the speaker can converse using only that visual information, it does not need to be converted into a speech signal by the speech synthesis unit.

[0013] The models that realize the speech recognition model, language model, and speech synthesis model may be trained using machine learning algorithms or realized by any other means, as long as they provide the desired input and output functions described above.

[0014] Voice signals input through the voice input unit are usually stored sequentially in a voice input buffer, and then transferred to the voice recognition unit, which is the next processing pipeline, when certain conditions are met. Similarly, the audio signal transferred to the audio output unit is temporarily stored in the audio output buffer, and when certain conditions are met, it is reproduced in the audio output unit, which is the next processing pipeline. These buffers may be omitted as long as they fulfill the purposes of the aforementioned voice input / output, voice recognition, and voice synthesis sections.

[0015] In a typical spoken dialogue system, the conditions for transferring speech data stored in the speech input buffer to the speech recognition unit are determined by end-of-utterance (EOU) detection. The EOU model is an input / output model for identifying when a speaker has finished speaking, and generally predicts the end of the utterance based on silent periods, prosodic information, or the content of the utterance. The EOU detector can be either independent or built into the speech recognition model. The longer the silent period input, the more accurately the EOU can predict the end of the utterance, but there is a trade-off with increased latency. For example, if a silent period of 1.5 seconds is detected, there is a high probability that the utterance has ended, but at the same time, it means that the delay until the response will naturally increase beyond 1.5 seconds. To eliminate the latency and uncertainty caused by this end-of-utterance detection, the voice signal is stored in a buffer while the trigger button is pressed, and when the button is released, the buffer is transferred to the next pipeline, the voice recognition unit. The trigger button can be any type of interface that allows the speaker to start and stop speech input, such as an interface where the button is pressed when a specific muscle in the body is pressed and released when the muscle is released.

[0016] A more specific embodiment will now be described. In particular, the details of how the aforementioned trigger button can be realized as an HTML element of a web browser and embedded in any website to provide the website with a voice dialogue function as an operator will be explained. On a website with an embedded trigger button, the voice signal captured from the voice input unit of the computer terminal is held in the voice input buffer only while the user is pressing the button. While pressing the button, the user speaks a question related to the website. For example, if it is an automobile manufacturer's website, the user might ask how much better the fuel economy of the car they are planning to replace is compared to the fuel economy of the car they currently own. When the user releases the trigger button, the voice signal stored in the voice input buffer is transferred to the voice recognition unit. Since end-of-utterance detection is omitted at this point, the time from the end of voice input to the voice signal being transferred to the voice recognition unit is simply the time it takes to transfer the voice signal. The voice signal input to the voice recognition unit is then passed through a text generation unit using a language model, a voice synthesis unit, and a voice output unit, where a response is played to the user. Generally, detecting the end of a speech takes around 0.5 to 1 second, and other processing takes more than 0.5 seconds, so the proportion of the overall delay caused by detecting the end of a speech is large. By omitting this step, it is possible to complete the process from the end of voice input to the playback of a response in less than 0.5 seconds.

[0000] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims.

Claims

1. In a speech dialogue system that runs on a computer terminal, It has a voice input unit, a voice input buffer, a voice recognition unit, a text generation unit using a language model, a voice synthesis unit, a voice output unit, and a trigger button. A voice dialogue system in which voice input from the voice input unit is stored in the voice buffer and then converted into text by the voice recognition unit, a response text to the input text is output by the text generation unit, the response text is then converted into a voice signal by the voice synthesis unit, and the voice signal is output as a physical sound wave from the voice output unit, While the trigger button is pressed, the voice signal is input to the voice input buffer, and when the trigger button is released, the voice signal stored up to that point is sent to the voice recognition unit, and subsequent processing is executed. To achieve high-speed response by eliminating the need for end-of-utterance detection processing and eliminating the delay associated with end-of-utterance detection processing; A voice dialogue system characterized by:

2. The speech dialogue system according to claim 1 The trigger button according to claim 1 can be configured using HTML elements that run on a web browser, and by installing this trigger button on any website, it is possible to provide a voice interaction function as an operator on the website. A website-embedded voice dialogue Q&A system.

Images