Dialogue system, dialogue processing method and electronic device

Abstract

The present invention relates to a dialogue system, a dialogue processing method, and an electronic device. The dialogue system, the dialogue system control method, and the electronic device are configured to process the user's voice before the user's voice ends to generate a system response by recognizing the user's intention. When the user's voice ends, the system response is output to continue the natural dialogue flow in real time. The dialogue system includes: a speech-to-text conversion engine that converts the user's voice into text; an intermediate dialogue engine that is configured to process the intermediate speech before the user's voice ends; and a final dialogue engine that is configured to process the final speech after the user's voice ends. The controller is configured to input the converted text to the intermediate dialogue engine when the user's voice has not ended, and to input the converted text to the final dialogue engine when the user's voice ends. The dialogue system also includes a text-to-speech conversion engine that is configured to convert the system response into a speech signal.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2019-0116007 filed on September 20, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0003] Embodiments of the present invention relate to a dialogue system, a method for controlling the dialogue system, and an electronic device capable of communicating with a user. Background Art

[0004] A dialog system is a system that can recognize a user's voice and provide a service corresponding to the recognized voice.

[0005] The dialog system may output a system response to provide a service corresponding to the recognized voice. The system response may include an answer to the user's question, a question for confirming the user's voice, or a guide to the service to be provided.

[0006] However, because the dialogue system starts recognition after the user's speech ends, there is a delay between the user's speech and the system response, making it difficult to continue a natural flow of dialogue. Summary of the Invention

[0007] One aspect of the present invention is to provide a dialogue system, a dialogue system control method, and an electronic device for processing a user's speech (i.e., user voice) in the dialogue system to recognize the user's intention (i.e., user intent) by using intermediate speech, thereby generating a system response before the user's speech ends. When the user's speech ends, the dialogue system, the dialogue system control method, and the electronic device are configured to output the generated system response, which can continue the natural flow of the dialogue without delay.

[0008] According to an embodiment of the present invention, a dialogue system includes: a speech to text conversion (STT) engine, an intermediate dialogue engine, a final dialogue engine, a controller and a text to speech conversion (TTS) engine, wherein the speech to text conversion (STT) engine is configured to convert the user voice input through a microphone into text in real time; the intermediate dialogue engine is configured to process the intermediate voice before the user voice is terminated to identify the user intention and generate a system response corresponding to the user intention; the final dialogue engine is configured to process the final voice after the user voice is terminated; the controller is configured to input the converted text to the intermediate dialogue engine when the user voice is not terminated, and input the converted text to the final dialogue engine when the user voice is terminated; the text to speech conversion (TTS) engine is configured to convert the system response into a voice signal.

[0009] In addition, before the user's voice is terminated, the intermediate dialogue engine can pre-generate a system response corresponding to the user's intention.

[0010] In addition, when the user voice is terminated, the controller can control the TTS engine to output a pre-generated system response.

[0011] In addition, when the intermediate conversation engine does not recognize the user intention, the controller may input the next user voice input through the microphone to the intermediate conversation engine.

[0012] In addition, the system may further include a storage device for storing the final speech and the user intention recognized by the final dialogue engine corresponding to the final speech when the user speech is terminated.

[0013] In addition, the controller may generate first intermediate voice data by deleting one word from the stored final voice, and may input the first intermediate voice data to the final dialogue engine.

[0014] In addition, when the user intention corresponding to the first intermediate voice data recognized by the final dialogue engine corresponds to the user intention corresponding to the final voice, the controller may store the first intermediate voice data in the storage device.

[0015] In addition, when the user intention corresponding to the first intermediate voice data recognized by the final dialogue engine corresponds to the user intention corresponding to the final voice, the controller may generate second intermediate voice data by deleting a word from the first intermediate voice data. The controller may input the second intermediate voice data into the final dialogue engine.

[0016] In addition, the controller may train the intermediate dialogue engine by using the stored first intermediate voice data.

[0017] In addition, the controller may generate rules used by the intermediate dialogue engine by using the stored first intermediate voice data.

[0018] According to an embodiment of the present invention, a conversation processing method includes: converting a user voice input through a microphone into text in real time; when the user voice has not terminated, inputting the converted text into an intermediate conversation engine that processes the intermediate voice; when the user voice terminates, inputting the converted text into a final conversation engine that processes the final voice; and generating a system response corresponding to the user intention recognized by the intermediate conversation engine or the user intention recognized by the final conversation engine.

[0019] Furthermore, generating the system response may include: when the intermediate dialogue engine recognizes the user intention, pre-generating the system response corresponding to the user intention before the user voice is terminated.

[0020] Furthermore, the method may include outputting a pre-generated system response when the user voice is terminated.

[0021] Furthermore, the method may include inputting a next user voice input through a microphone to the intermediate conversation engine when the intermediate conversation engine does not recognize the user intention.

[0022] Furthermore, the method may include, when the user voice is terminated, storing a final voice and a user intention recognized by a final dialogue engine corresponding to the final voice.

[0023] Furthermore, the method may include generating first intermediate voice data by deleting one word from the stored final voice, and inputting the first intermediate voice data to the final dialogue engine.

[0024] Furthermore, the method may include storing the first intermediate voice data in a storage device when the user intention corresponding to the first intermediate voice data recognized by the final dialogue engine corresponds to the user intention corresponding to the final voice.

[0025] In addition, the method may include: when the user intention corresponding to the first intermediate voice data recognized by the final dialogue engine corresponds to the user intention corresponding to the final voice, generating second intermediate voice data by deleting a word from the first intermediate voice data, and inputting the second intermediate voice data into the final dialogue engine.

[0026] Furthermore, the method may include training the intermediate dialogue engine by using the stored first intermediate speech data.

[0027] Furthermore, the method may include generating rules used by the intermediate dialogue engine by using the stored first intermediate speech data.

[0028] According to one embodiment, an electronic device includes: a memory configured to store one or more programs; and a processor configured to process the one or more stored programs. The one or more programs may include: converting user speech input through a microphone into text in real time; inputting the converted text into an intermediate dialogue engine that processes intermediate speech when the user speech has not terminated; inputting the converted text into a final dialogue engine that processes final speech when the user speech has terminated; and generating a system response corresponding to the user intent recognized by the intermediate dialogue engine or the user intent recognized by the final dialogue engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] These and / or other aspects of the present invention will become more apparent and more readily appreciated from the following description of embodiments presented in conjunction with the accompanying drawings, in which:

[0030] Figure 1 is a control block diagram of a dialog system according to an implementation scheme.

[0031] Figure 2 is a diagram illustrating an example of a dialog system for processing user speech according to an embodiment.

[0032] Figure 3 and Figure 4 An example of a database stored in a storage device of a dialog system is shown.

[0033] Figure 5 and Figure 6 An example of data processing for constructing an intermediate dialog engine is shown.

[0034] Figure 7 Another example of a database stored in the storage device of the dialog system is shown.

[0035] Figure 8 is a flowchart of a conversation processing method according to an implementation scheme.

[0036] Figure 9 is a flow chart illustrating a process of updating an intermediary conversation engine using data stored in a log database (DB), according to an embodiment. DETAILED DESCRIPTION

[0037] The embodiments described herein and the configurations shown in the drawings illustrate various examples of the present invention. At the time of filing this application, there may be various modifications that can replace the embodiments and drawings of this specification.

[0038] Furthermore, the terms used herein are intended only to describe specific embodiments and are not intended to limit the present invention. Unless the context clearly indicates otherwise, singular expressions include plural expressions. As used herein, the terms "including" or "having" are intended to indicate the presence of the features, values, steps, actions, components, parts, or combinations thereof described in this specification. The terms do not exclude the presence or addition of one or more other features or values, steps, actions, components, parts, or combinations thereof.

[0039] In addition, terms such as "component," "group," "block," "member," and "module" may refer to a unit for processing at least one function or operation. For example, these terms may refer to at least one hardware (such as a field programmable gate array (FPGA) / application-specific integrated circuit (ASIC)), at least one software stored in a memory, or at least one hardware processed by a processor.

[0040] In addition, ordinal numbers such as "first" and "second" used before components described herein are only used to distinguish components from each other. Ordinal numbers used before components are not used to specify the order in which these components are connected or used. Ordinal numbers do not have a different meaning, such as priority.

[0041] The reference numerals accompanying the steps are used to identify the steps. These reference numerals do not indicate the order between the steps. Unless the context clearly indicates a specific order, each step can be performed in an order different from the order described.

[0042] On the other hand, the disclosed embodiments can be implemented in the form of a recording medium for storing computer-executable instructions. The instructions can be stored in the form of program code, and when executed by a processor, a program module can be generated to perform the operations of the disclosed embodiments. The recording medium can be implemented as a computer-readable recording medium.

[0043] The computer-readable recording medium may include various recording media having stored therein instructions that can be read by a computer, such as read-only memory (ROM), random access memory (RAM), magnetic tapes, magnetic disks, flash memory, optical data storage devices, and the like.

[0044] Furthermore, when an element in the written description and claims is described as “configured to” perform or implement a stated function, step, instruction set, etc., the element can also be considered to be “configured to” do so.

[0045] Hereinafter, embodiments of a dialog system, a dialog processing method, and an electronic device according to an aspect are described in detail with reference to the accompanying drawings.

[0046] Figure 1 This is a control block diagram of the dialogue system according to the implementation scheme. Figure 1 According to the embodiment, the dialogue system 100 may include: a speech to text (STT) engine 110, an intermediate dialogue engine 130, a final dialogue engine 140, a controller 120, a storage device 150 and a text to speech (TTS) engine 160; the speech to text conversion (STT) engine 110 is configured to convert the user voice input through the microphone 10 into text in real time; the intermediate dialogue engine 130 is configured to process the intermediate voice before the user voice is terminated to recognize the user intention and generate a system response corresponding to the user intention; the final dialogue engine 140 is configured to process the final voice after the user voice is terminated; the controller 120 is configured to input the converted text to the intermediate dialogue engine 130 when the user voice is not terminated, and to input the converted text to the final dialogue engine 140 when the user voice is terminated; the storage device 150 is configured to store various data required for the operation of the dialogue system 100; the text to speech conversion (TTS) engine 160 is configured to convert the system response into a voice signal. The system response converted into a speech signal can be output through the speaker 20 .

[0047] According to an embodiment, the dialogue system 100 may be implemented as a server. In this case, the components 110, 120, 130, 140, 150, and 160 of the dialogue system 100 may be provided in the server. However, some of the components 110, 120, 130, 140, 150, and 160 of the dialogue system 100 may also be provided in a user terminal that connects the dialogue system 100 to a user.

[0048] According to an embodiment, the dialog system 100 may include at least one memory for storing programs and related data, and at least one processor for executing the stored programs.

[0049] The STT engine 110, controller 120, intermediate dialogue engine 130, final dialogue engine 140 and TTS engine 160 may also use separate memories and processors, or some of the STT engine 110, controller 120, intermediate dialogue engine 130, final dialogue engine 140 and TTS engine 160 may share memories and processors.

[0050] In addition, the controller 120 may be provided in a separate configuration. Operations of the controller 120, which will be described later, may be performed in a configuration related to corresponding operations in the STT engine 110, the intermediate dialogue engine 130, the final dialogue engine 140, and the TTS engine 160. For example, operations related to control of the intermediate dialogue engine 130 during the operation of the controller 120 may be performed by the intermediate dialogue engine 130.

[0051] In other words, the components in this embodiment are categorized based on operation rather than actual location or configuration. Therefore, even if the dialog system does not have a separate processor called a controller, the operations of the controller 120 described herein can refer to the operations of a processor called a dialog engine without departing from the scope of the dialog system 100 according to the embodiment.

[0052] In addition, in some cases, the dialogue system may not be provided with an intermediate dialogue engine and a final dialogue engine separately. In this case, as described throughout the present invention, the dialogue engine performs the operations of the intermediate dialogue engine 130 and the final dialogue engine 140. In this embodiment, the dialogue engine is included in the scope of the dialogue system (100). In other words, the program that performs the operations of the intermediate dialogue engine 130 and the program that performs the operations of the final dialogue engine 140 can be executed in one processor, or the operations of the intermediate dialogue engine 130 and the operations of the final dialogue engine 140 can both be executed in one program.

[0053] For example, when the user terminal is a vehicle, some components of the dialogue system 100 may be provided in the vehicle. When the user terminal is a mobile device (such as a smartphone), an intelligent artificial intelligence (AI) speaker, or a PC, some components of the dialogue system 100 may be provided in the mobile device, the AI ​​speaker, or the PC.

[0054] Alternatively, the user may download and use a program for executing some functions of the components of the dialogue system 100 on the user terminal.

[0055] The microphone 10 and the speaker 20 may be provided in a user terminal such as a vehicle, a mobile device, or a PC. The user terminal may be connected to the dialogue system 100 via wireless communication. Figure 1 Although not shown in the control block diagram, the dialogue system 100 may further include a wireless communication module, which can exchange data with the user terminal through wireless communication.

[0056] The user voice input through the microphone 10 may be transmitted to the wireless communication module of the dialogue system 100 in the form of a stream signal.

[0057] The STT engine 110 can convert speech into text by applying a speech recognition algorithm to the user speech transmitted in the form of a stream signal. For example, by applying a feature vector extraction technique (such as Cepstrum, Linear Predictive Coefficient (LPC), Mel Frequency Cepstral Coefficient (MFCC), or Filter Bank Energy), a feature vector of a speech signal corresponding to the user speech can be extracted.

[0058] Recognition results can be obtained by comparing the extracted feature vectors with trained reference patterns. To this end, an acoustic model and a language model can be used. The acoustic model is used to model and compare the signal characteristics of the speech, and the language model is used to model the linguistic order relationship such as the words or syllables corresponding to the recognized vocabulary.

[0059] Acoustic models can be further divided into direct comparison methods and statistical methods. Direct comparison methods set the recognition object as a feature vector model and compare it with the feature vector of the speech data. Statistical methods perform statistical processing on the feature vector of the recognition object.

[0060] The direct comparison method uses a feature vector model for units such as words or phonemes to be recognized and compares the similarity of the input speech. Using the vector quantization method, the feature vectors of the input speech data are mapped to a codebook serving as a reference model and encoded into representative values, thereby comparing the code values.

[0061] Statistical modeling is a method that structures the units of an identified object into a sequence of states and uses the relationships between these sequences. A sequence of states can include multiple nodes. Other methods that use relationships between state sequences include dynamic time warping (DTW), hidden Markov models (HMMs), and neural networks.

[0062] Dynamic Time Warping (DTC) is a technology that compensates for timeline differences while taking into account the dynamic nature of speech. Speech length varies over time, even when the same utterance is produced by the same speaker. The HMM assumes speech is a Markov process with state transition probabilities and observation probabilities for nodes (output symbols) in each state. The HMM then estimates these state transition probabilities and node observation probabilities based on training data. Within this estimated model, the HMM, as a recognition technique, calculates the probability that the input speech will occur.

[0063] On the other hand, language models, which are used to model linguistic sequential relationships such as words or syllables, can apply the acoustic relationships between language units to the units obtained in speech recognition, thereby reducing acoustic ambiguity and reducing recognition errors. Language models are based on statistical language models and finite state automata (FSA). Statistical language models use chain probabilities of words, such as unigrams, bigrams, and trigrams.

[0064] The STT engine 110 may use any of the above methods when recognizing a speech signal corresponding to a user's speech and converting it into text. For example, an acoustic model with a hidden Markov model may be used, or an N-best search method that integrates an acoustic model and a speech model may be used. The N-best search method can improve recognition performance by selecting up to N recognition result candidates using an acoustic model and a language model, and then re-evaluating the ranking of these candidates.

[0065] In addition, the STT engine 110 can calculate a confidence score to ensure the reliability of the recognition result. For speech recognition results, the confidence score is an indicator of the reliability of the result. The confidence score can be defined as the relative value of the phoneme or word as the recognition result and the probability of saying the word from other phonemes or words. Therefore, the confidence score can be expressed as a value between 0 and 1, or as a value between 0 and 100. If the confidence score is greater than a predetermined threshold, the recognition result can be recognized, and if the confidence score is smaller, the recognition result can be rejected.

[0066] The STT engine 110 can determine whether the user's speech has terminated by applying an endpoint detection (EPD) algorithm to the speech signal corresponding to the user's speech. Various EPD algorithms are known in the field of speech recognition. The STT engine 110 can use any known algorithm.

[0067] Regardless of whether the user voice is terminated, the STT engine 110 can convert the user voice input to the microphone 10 into text in real time and output the text. In this case, EPD can be performed together. The real-time converted text and EPD result can be output together.

[0068] The intermediate dialogue engine 130 and the final dialogue engine 140 can apply natural language understanding to the text output from the STT engine 110 to understand the user's intention corresponding to the user's voice. Hereinafter, the description generally applied to the intermediate dialogue engine 130 and the final dialogue engine 140 is collectively referred to as a dialogue engine.

[0069] The conversation engine performs stemming on the user's speech in text form, converting the input string into a sequence of stems. The conversation engine can also identify entity names from the user's speech. Entity names are proper nouns such as names of people, places, organizations, times, dates, and currencies. Entity name recognition is the task of identifying entity names in a sentence and determining the type of entity name identified. By identifying individual names, we can extract important keywords from the sentence to understand its meaning.

[0070] The conversation engine can extract domains from the user's speech. A domain can identify the subject of the language spoken by the user. For example, domains representing various topics (such as text, navigation, schedule, weather, traffic, vehicle control, etc.) can be stored in a database in storage device 150.

[0071] The conversation engine can analyze the voice behavior of the user's voice. Voice behavior analysis is configured to analyze the intent of the user's voice and understand the user's voice intent, such as whether the user is asking a question, making a request, responding, or expressing simple emotions.

[0072] The conversation engine can grasp the user's intention based on information such as the domain corresponding to the user's voice, entity name, voice behavior, etc., and can extract actions corresponding to the user's intention. Actions can be defined by objects and operators.

[0073] For example, when the recognized user intent is "run the air conditioner", the corresponding action can be defined as: "air conditioner (object) _ on (operator)". When the recognized user intent is "send a text message", the corresponding action can be defined as: "text (object) _ send (operator)".

[0074] In addition, the dialog engine can extract factors related to the execution of the action. The factors related to the execution of the action can be effective factors directly required for the execution of the action, or can be invalid factors used to extract such effective factors.

[0075] For example, if the text output by the STT engine 110 is "Send a text message to Jidong", the dialogue engine determines that the domain corresponding to the user's voice is "text", the action is "send text", and the voice behavior corresponds to "request".

[0076] The object name "Jidong" corresponds to {factor 1: recipient}, which is relevant for executing the action. However, to actually send the text message, {factor 2: specific message content} is required. In this case, the dialogue system 100 can output a system response: "Please tell me the message to send" to obtain the specific message content from the user.

[0077] On the other hand, the difference between the intermediate dialogue engine 130 and the final dialogue engine 140 is that the intermediate dialogue engine 130 processes the user's speech before it is terminated. In other words, the intermediate dialogue engine 130 can determine the user's intention by analyzing the sentence, even if it is not a complete sentence processed by applying general natural language understanding technology.

[0078] To this end, the dialogue system 100 may learn the intermediate speech processing as described below, or may establish rules for the intermediate speech processing.

[0079] When the user speech is not finished, the controller 120 inputs the text output from the STT engine 110 (i.e., the text of the intermediate speech) to the intermediate dialogue engine 130. When the user speech is finished, the text output from the STT engine 110 (i.e., the text of the final speech) may be input to the final dialogue engine 140.

[0080] As described above, since the STT engine 110 outputs the EPD result together with the text corresponding to the user voice, the controller 120 may determine whether the user voice is terminated based on the EPD result output from the STT engine 110 .

[0081] As described above, intermediate dialogue engine 130 and final dialogue engine 140 analyze the input intermediate and final speech, respectively, to understand the user's intent. If the user's intent is successfully recognized, a result may be output. The output of intermediate dialogue engine 130 and final dialogue engine 140 may include an action corresponding to the user's intent, factors related to the action, and a system response corresponding to the user's intent.

[0082] The intermediate dialogue engine 130 can generate a system response corresponding to the user's intention. For example, if the user's intention is to run the air conditioner, the intermediate dialogue engine 130 can generate a system response "Do you want to run the air conditioner?" If the user's intention is to send a text message, the intermediate dialogue engine 130 can generate a system response "Who to send the text message to?"

[0083] However, if the intermediate dialogue engine 130 cannot identify the user's intent corresponding to the intermediate speech, the final dialogue engine 140 can identify the user's intent corresponding to the final speech and generate a system response corresponding to the user's intent. The generated system response is input to the TTS engine 160. The TTS engine 160 can convert the system response into a speech signal.

[0084] The intermediate dialogue engine 130 can generate a system response corresponding to the user's intention before the user's voice is terminated. In addition, since the intermediate dialogue engine 130 processes the intermediate voice to generate the system response before the user's voice is terminated, the TTS engine 110 can also convert the system response into a voice signal before the user's voice is terminated.

[0085] In addition, the controller 120 can control the TTS engine 110 to output a system response generated immediately after the user voice is terminated. Since the delay between the user voice and the system response is reduced, the user can feel as if they are actually having a conversation, and the practicality of the conversation system can be improved.

[0086] Figure 2 is a diagram illustrating an example of a dialog system for processing user speech according to an embodiment.

[0087] Reference Figure 2 , the following describes the operation of the dialogue system 100 by taking the user's final voice "Dalee! Please turn on the air conditioner" as an example. In this example, "Dalee" is the wake-up word of the dialogue system (100).

[0088] When a user voice is input through the microphone 10, the STT engine 110 converts the input user voice into text in real time. If the STT engine 110 inputs "Dalee", the STT engine 110 can convert it into text and recognize that a wake-up word has been input. When the wake-up word is recognized, the dialogue system (100) is activated.

[0089] If "air conditioning" is input after the wake-up word "Dalee", the STT 110 engine can convert it into text and output the text together with the EPD result (not the end of the voice). Thus, "air conditioning" becomes the middle voice.

[0090] The controller 120 may input the intermediate voice "air conditioning" into the intermediate dialogue engine 130. When the intermediate dialogue engine 130 processes "air conditioning", it is difficult to determine the user's intention. Therefore, the processing result is failure.

[0091] When the intermediate conversation engine 130 cannot recognize the user intention, in other words, when the processing result of the intermediate conversation engine 130 fails, the controller 120 may input the next user voice input through the microphone 10 to the intermediate conversation engine 130 .

[0092] In this example, the next user voice input may be input into the intermediate dialogue engine 130 until the next air conditioner “turn on.” The intermediate dialogue engine 130 may apply natural language understanding technology to identify the user intent corresponding to the intermediate voice “turn on the air conditioner.”

[0093] The intermediate dialogue engine 130 may recognize the user intent corresponding to the intermediate speech "turn on the air conditioner" as "run the air conditioner" and generate a system response corresponding to the user intent. In this example, the system response "run the air conditioner" may be generated to confirm the user intent and guide the provided service.

[0094] If the next user voice "Would you like to" is input and the EPD result included in the output of the STT engine 110 indicates the end of the voice, the controller 120 may input the final voice "Please turn on the air conditioner" to the final dialogue engine 140. The processing result of the final dialogue engine 140 may be stored in the storage device 150 as described below.

[0095] In addition, when the user voice is terminated, the controller 120 may control the TTS engine 160 and output a pre-generated service response.

[0096] On the other hand, in the example, when the user's intention is successfully determined using the intermediate speech as described above, there may be a situation where the user's intention cannot be determined using the intermediate speech. In this case, a system response corresponding to the user's intention identified in the final dialogue engine 140 can be generated.

[0097] Figure 3 、 Figure 4 and Figure 7 An example of a database (DB) stored in a storage device of a dialogue system is shown. Figure 5 and Figure 6 An example of data processing for constructing an intermediate dialog engine is shown.

[0098] The dialogue system 100 according to the embodiment may use a rule-based dialogue processing algorithm or a learning-based dialogue processing algorithm. In the case of using a rule-based dialogue processing algorithm, such as Figure 3 As shown, rules for dialog processing are stored.

[0099] Reference Figure 3 The storage device 150 may include an intermediate rule DB 151 for the dialogue process of the intermediate dialogue engine 130 and a final rule DB 152 for the dialogue process of the final dialogue engine 140 .

[0100] The storage device 150 may include a non-volatile memory such as a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EPROM), a flash memory, etc. The storage device 150 may include a volatile memory such as a static random access memory (S-RAM) and a dynamic random access memory (D-RAM). Depending on the type of information to be stored, the information may be stored in a non-volatile memory or in a volatile memory.

[0101] The intermediate rule DB 151 may store rules for recognizing a user intention included in an intermediate speech, rules for generating a system response corresponding to the user intention, and the like.

[0102] Furthermore, the final rule DB 152 may store rules for recognizing a user intention included in the final speech, rules for generating a system response corresponding to the user intention, and the like.

[0103] The processing result of the final dialogue engine can be used to construct or update the intermediate dialogue engine 130. Figures 4 to 6 Description.

[0104] Reference Figure 4 The storage device 150 may include a log DB 153 for storing log data of a final dialogue engine and an intermediate dialogue engine DB 154 for storing data used to construct or update an intermediate dialogue engine.

[0105] The log data may include the user's speech in text form and the user's intent recognized by the final dialog engine 140 .

[0106] like Figure 5 As shown, the user intent determined by the final dialogue engine 140 can be matched with the user speech in text form input to the final dialogue engine 140. For example, the user speech "turn on the air conditioner" can be matched with the corresponding user intent "air conditioner_on" and stored. The user speech "turn off the air conditioner" can be matched with the corresponding user intent "air conditioner_off" and stored. The user speech "set the temperature of the air conditioner to 18 degrees" can be matched with the user intent "air conditioner_set (18°C)" and stored.

[0107] The controller 120 processes the log data stored in the log DB 153 to generate data for constructing or updating the intermediate conversation engine 130 .

[0108] For example, intermediate dialogue data may be generated by deleting a word from the user's speech stored in the log DB 153. The generated intermediate dialogue data may be input to the final dialogue engine 140. If the user's intention determined by the final dialogue engine 140 matches the user's intention stored in the log DB 153, the intermediate dialogue data may be stored in the intermediate dialogue engine DB 154.

[0109] The above process may be repeated by deleting one more word until the dialog engine 140 is ultimately unable to recognize the user's intention, or until the user's intention determined by the dialog engine 140 ultimately does not match the user's intention stored in the log DB 153 .

[0110] Specifically, first intermediate speech data may be generated by deleting one word from the user speech stored in the log DB 153. The first intermediate speech data may be input to the final dialogue engine 140. If the user intent corresponding to the first intermediate speech data grasped by the final dialogue engine 140 matches the user intent stored in the log DB 153, the first intermediate speech data may be stored in the intermediate dialogue engine DB 154.

[0111] Specifically, first intermediate voice data may be generated by deleting one word from the user voice stored in the log DB 153. The first intermediate voice data may be input to the final dialogue engine 140. If the user intent corresponding to the first intermediate voice data identified by the final dialogue engine 140 matches the user intent stored in the log DB 153, the first intermediate voice data may be stored in the intermediate dialogue engine DB 154.

[0112] The intermediate conversation engine 130 may be updated by the intermediate speech data stored in the intermediate conversation engine DB 154. In this example, the updating may include learning or generating new rules using the stored data.

[0113] The controller 120 may further generate second intermediate voice data by deleting a word from the first intermediate voice data. The controller 120 may input the second intermediate voice data into the final dialogue engine 140. If the user intent corresponding to the second intermediate voice data identified by the final dialogue engine 140 matches the user intent stored in the log DB 153, the second intermediate voice data may also be stored in the intermediate dialogue engine DB 154. The intermediate dialogue engine 130 may be updated again with the added data.

[0114] Eventually, the dialogue engine 140 may repeat the above process until the user's intention cannot be understood or the user's intention is different from the user's intention stored in the log DB 154 .

[0115] Reference Figure 6 , when the first intermediate voice data “turn on the air conditioner” generated by deleting a word (character) from the user voice “turn on the air conditioner” stored in the log DB 153 is input into the final dialogue engine 140, if the user intention output from the final dialogue engine 140 is “air conditioner_on”, the controller 120 matches the first intermediate voice data “turn on the air conditioner” with the user intention “air conditioner_on” and stores the result in the intermediate dialogue engine DB 154.

[0116] Finally, the dialogue engine 140 outputs the first intermediate voice data "turn off air conditioning" generated by deleting one word (character) from the user voice "turn off air conditioning" stored in the log DB 153. If the user intention is "air conditioning_off", the controller 120 may match the first intermediate voice data "turn off air conditioning" with the user intention "air conditioning_off" and store the first intermediate voice data "turn off air conditioning" in the intermediate dialogue engine DB 154.

[0117] In addition, when the first intermediate voice data "Please set the temperature of the air conditioner to 18" generated by deleting a word (character) from the user voice "Please set the temperature of the air conditioner to 18 degrees" stored in the log DB 153 is input into the final dialogue engine 140, if the user intention output from the final dialogue engine 140 is "Air Conditioning_Setting (18℃)", the controller 120 matches the first intermediate voice data "Please set the temperature of the air conditioner to 18" with the user intention "Air Conditioning_Setting (18℃)" stored in the dialogue engine DB 154.

[0118] In an example of a rule used by the intermediate dialogue engine 130 to recognize the user's intention, a lexical semantic model may be used. Therefore, the controller 120 may generate a lexical semantic model based on the intermediate speech data stored in the intermediate dialogue engine DB 154.

[0119] Furthermore, when generating rules, the data in the intermediate speech data that the final dialogue engine 140 failed to recognize as matching the user's intention can be used. Therefore, in this case, the data in the intermediate speech data (one word (character) has been deleted from the user's speech) that the final dialogue engine 140 failed to recognize as matching the user's intention can be stored in a separate rule generation DB 155, such as Figure 7 In this case, the data obtained by differently recognizing the user's intention is not used.

[0120] The rules used by the intermediate dialogue engine 130 can also be manually created by the user. Specifically, by directly identifying the user's intention, the data that the final dialogue engine 140 fails to identify as matching the user's intention stored in the rule generation DB 155 can generate rules.

[0121] For example, if the user's speech recorded in log database 153 is "Please run the air conditioner," final dialogue engine 140 may recognize that the first intermediate speech data "Run the air conditioner" has one word (character) deleted. Accordingly, the second intermediate speech data "Air conditioner" recorded in intermediate dialogue engine database 154 has another word deleted. In this example, final dialogue engine 140 cannot understand the user's intention and stores the second intermediate speech data in rule generation database 155.

[0122] In another example, if the user's speech recorded in log database 153 is "Please turn the air conditioner downward," final dialogue engine 140 may recognize the first intermediate speech data, which has one word deleted, as "turn the air conditioner downward." Consequently, final dialogue engine 140 may not recognize the second intermediate speech data, which has another word deleted, as "turn the air conditioner downward," as matching the user's intention. Therefore, the second intermediate speech data may be stored in rule generation database 155.

[0123] The user can manually create a rule by using the intermediate speech data stored in the rule generation DB 155. For example, if the user manually recognizes the intention, they can understand "direction of air conditioner." Therefore, the following rule can be generated: <User speech: direction of air conditioner → User intention: "air conditioner_direction (downward)">.

[0124] On the other hand, since "air conditioning" stored in the rule generation DB 155 cannot be manually understood, "air conditioning" is not used for rule generation.

[0125] Hereinafter, a dialog processing method according to an embodiment is described. The dialog system according to the above embodiment can be used to implement the dialog processing method according to the embodiment. Therefore, even if not mentioned otherwise, reference is made to Figures 1 to 7 The description is also applicable to the embodiment of the dialog processing method. The following description of the embodiment of the dialog processing method is also applicable to the embodiment of the dialog system.

[0126] Figure 8 is a flowchart of a conversation processing method according to an implementation scheme.

[0127] according to Figure 8 The dialogue processing method shown, when the user's voice is input through the microphone 10 ("Yes" in 310), converts the user's voice into text in real time (311).

[0128] As described above, when the user voice is input to the STT engine 110 in the form of a stream signal, the STT engine 110 can convert the input user voice into text in real time and output the text. At this time, the EPD result can also be output.

[0129] If the user's speech has not ended ("No" in 312), the converted text is input into the intermediate dialogue engine 130 (313). If the user's speech has ended ("Yes" in 312), the converted text is input into the final dialogue engine 140 (316).

[0130] The controller 120 may determine whether to end the user speech based on the EPD result output from the STT engine 110. Therefore, if the EPD result indicates that the user speech is not the end of the user speech, the converted text (i.e., the text of the intermediate speech) is input into the intermediate dialogue engine 130. If the EPD result indicates that it is the end of the user speech, the converted text (i.e., the text of the final speech) is input into the final dialogue engine 140.

[0131] If the intermediate dialogue engine 130 has recognized the input text, that is, the user intention corresponding to the intermediate speech ("Yes" in 314), the system generates a system response (315) corresponding to the user intention recognized by the intermediate dialogue engine 130 before the end of the user speech. If the intermediate dialogue engine 130 has not recognized the user intention corresponding to the intermediate speech, the text of the next user speech input through the microphone 10 is input to the intermediate dialogue engine 130.

[0132] If the intermediate dialogue engine 130 has identified the user's intention corresponding to the intermediate speech, a system response may be generated in advance before the user's utterance is terminated.

[0133] When the user speech ends, the pre-generated system response is output (317). If the intermediate dialogue engine fails to recognize the user intention using the intermediate speech, the final dialogue engine 140 generates a system response corresponding to the user intention determined using the final speech (315).

[0134] The TTS engine 317 can convert the generated system response into a voice signal. The system response converted into a voice signal can be output to the user through the speaker 20 provided in the user terminal.

[0135] On the other hand, when the user voice is long, the user voice may continue even if the generation of the system response is completed. The controller 120 may control the TTS engine 160 to keep outputting the system response until the user voice is terminated, and output the system response after the user voice is terminated.

[0136] Furthermore, the user intention and the final speech recognized by the final dialogue engine 140 are stored in the log DB 153, and are used to update the intermediate dialogue engine 130. Hereinafter, a process of updating the intermediate dialogue engine 130 is described below.

[0137] Figure 9 is a flowchart illustrating a process of updating an intermediate conversation engine using data stored in a log DB according to an embodiment.

[0138] According to the above-described embodiment, the log DB 153 stores the final speech and the corresponding user intention.

[0139] according to Figure 9 In the dialog processing method shown, the controller 120 generates first intermediate speech data (320) by deleting one word from the final speech stored in the log DB 153, and inputs the first intermediate speech data to the final dialog engine 140.

[0140] When the final dialogue engine 140 recognizes the user intention corresponding to the first intermediate voice data ("Yes" in 322), the controller 120 determines whether the user intention corresponding to the first intermediate data matches the user intention corresponding to the final voice stored in the log DB (323). If the user intention corresponding to the first intermediate voice data matches the user intention corresponding to the final voice stored in the log DB ("Yes" in 324), the first intermediate voice data is stored in the intermediate dialogue engine DB 154 (325).

[0141] In addition, the controller 120 deletes a word from the first intermediate voice data to generate second intermediate voice data, and inputs the second intermediate voice data into the final dialogue engine. The above process of identifying the user's intention is repeated until the voice data does not match the user's intention corresponding to the final voice.

[0142] The intermediate conversation engine may learn by using the intermediate voice data stored in the intermediate conversation engine DB 154 , thereby updating the controller 120 .

[0143] On the other hand, when the final dialogue engine 140 fails to recognize the user intention corresponding to the first intermediate speech data (No in 322 ), the first intermediate speech data may be stored in the rule generation DB 155 ( 326 ).

[0144] The controller 120 may generate rules for the intermediate dialogue engine 130 by using the first intermediate speech data stored in the rule generation DB 155. A detailed description of rule generation is provided in the above-described embodiment of the dialogue system 100.

[0145] According to one embodiment, an electronic device includes: a memory configured to store one or more programs; and a processor configured to process the one or more stored programs. The one or more programs may include: converting user speech input through a microphone into text in real time; inputting the converted text into an intermediate dialogue engine that processes intermediate speech when the user speech has not terminated; inputting the converted text into a final dialogue engine that processes final speech when the user speech has terminated; and generating a system response corresponding to the user intent recognized by the intermediate dialogue engine or the user intent recognized by the final dialogue engine.

[0146] In addition, the program stored in the memory of the electronic device is used to execute the above-mentioned conversation processing method. Each step included in the above-mentioned conversation processing method can be further included in the program executed by the processor.

[0147] As described above, the disclosed embodiments have been described with reference to the accompanying drawings. It should be understood by those skilled in the art that the present invention may be implemented in forms different from the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are examples and should not be construed as limiting.

Claims

1. A dialogue system comprising: A speech-to-text conversion engine configured to convert user speech input through a microphone into text in real time; an intermediate dialogue engine configured to process the intermediate speech before the user speech is terminated to identify the user intent and generate a system response corresponding to the user intent; a final dialogue engine configured to process a final speech after the user speech is terminated; a controller configured to: input the converted text to the intermediate dialogue engine when the user voice is not terminated, and input the converted text to the final dialogue engine when the user voice is terminated; a text-to-speech engine configured to convert the system response into a speech signal; a storage device configured to store, when the user speech is terminated, a final speech and a user intention corresponding to the final speech and recognized by the final dialogue engine; The controller updates the intermediate dialogue engine based on the final speech and user intention stored in the storage device.

2. The dialogue system according to claim 1, wherein: Before the user's voice is terminated, the intermediate dialogue engine pre-generates a system response corresponding to the user's intention.

3. The dialogue system according to claim 2, wherein: When the user's voice is terminated, the controller controls the text-to-speech engine to output a pre-generated system response.

4. The dialogue system according to claim 1, wherein: When the intermediate conversation engine does not recognize the user intention, the controller inputs the next user voice input through the microphone to the intermediate conversation engine.

5. The dialogue system according to claim 1, wherein: The controller generates first intermediate voice data by deleting one word from the stored final voice, and inputs the first intermediate voice data to the final dialogue engine.

6. The dialogue system according to claim 5, wherein: When the user intention corresponding to the first intermediate voice data recognized by the final dialogue engine corresponds to the user intention corresponding to the final voice, the controller stores the first intermediate voice data in the storage device.

7. The dialogue system according to claim 6, wherein: When the user intention corresponding to the first intermediate voice data recognized by the final dialogue engine corresponds to the user intention corresponding to the final voice, the controller generates second intermediate voice data by deleting a word from the first intermediate voice data and inputs the second intermediate voice data to the final dialogue engine.

8. The dialogue system according to claim 6, wherein: The controller trains the intermediate dialogue engine by using the stored first intermediate voice data.

9. The dialogue system according to claim 6, wherein: The controller generates a rule used by the intermediate dialogue engine by using the stored first intermediate voice data.

10. A method for processing a conversation, the method comprising: Convert user voice input through microphone into text in real time; When the user's voice is not terminated, the converted text is input into an intermediate dialogue engine for processing the intermediate voice; When the user's speech is terminated, the converted text is input into the final dialogue engine that processes the final speech; generating a system response corresponding to the user intent identified by the intermediate dialogue engine or the user intent identified by the final dialogue engine; When the user speech is terminated, storing the final speech and the user intention corresponding to the final speech and recognized by the final dialogue engine in the storage device; The intermediate dialog engine is updated based on the final speech and the user intent stored in the storage device.

11. The method according to claim 10, wherein: Generating the system response includes: when the intermediate dialogue engine recognizes the user intention, pre-generating the system response corresponding to the user intention before the user voice is terminated.

12. The method according to claim 11, further comprising: When the user's voice ends, the pre-generated system response is output.

13. The method according to claim 12, further comprising: When the intermediate dialogue engine does not recognize the user intention, the next user voice input through the microphone is input to the intermediate dialogue engine.

14. The method according to claim 10, further comprising: First intermediate speech data is generated by deleting one word from the stored final speech, and the first intermediate speech data is input to the final dialogue engine.

15. The method according to claim 14, further comprising: When the user intention corresponding to the first intermediate voice data recognized by the final dialogue engine corresponds to the user intention corresponding to the final voice, the first intermediate voice data is stored in a storage device.

16. The method according to claim 15, further comprising: When the user intention corresponding to the first intermediate voice data recognized by the final dialogue engine corresponds to the user intention corresponding to the final voice, second intermediate voice data is generated by deleting a word from the first intermediate voice data, and the second intermediate voice data is input to the final dialogue engine.

17. The method of claim 15, further comprising: The intermediate dialogue engine is trained by using the stored first intermediate speech data.

18. The method of claim 15, further comprising: Rules used by the intermediate dialogue engine are generated by using the stored first intermediate speech data.

19. An electronic device comprising: a memory configured to store one or more programs; and a processor configured to process one or more stored programs; Wherein, the one or more programs include: Convert user voice input through microphone into text in real time; When the user's voice is not terminated, the converted text is input into an intermediate dialogue engine for processing the intermediate voice; When the user's speech is terminated, the converted text is input into the final dialogue engine that processes the final speech; generating a system response corresponding to the user intent identified by the intermediate dialogue engine or the user intent identified by the final dialogue engine; When the user speech is terminated, storing the final speech and the user intention corresponding to the final speech and recognized by the final dialogue engine in the storage device; The intermediate dialog engine is updated based on the final speech and the user intent stored in the storage device.

Images